Image acquisition device, method, and program

Abstract

An image acquisition method comprises: a display step in which a display unit 5 presents a presentation image having arranged therein, at different positions, presentation book images having the same motif but different art styles; an input reception step in which an input reception unit 6 receives a selective input operation for selecting a presentation book image having an art style preferred by an object person from among a plurality of presentation book images included in the presentation image; and an image selection control step in which an image selection control unit 4 obtains, as a search query image, a presentation book image corresponding to the selection input operation received by the input reception unit from among the presentation book images included in the presentation image.

Description

Image acquisition apparatus, method, and program

[0001] The disclosure technology relates, for example, to a technology for searching for books such as picture books with an art style preferred by a target audience, such as young children.

[0002] Picture books play an important role in a person's development during early childhood. For example, if it were possible to search for picture books that a target person, such as a young child, would like, the person would read the picture books found in the search results, which would encourage their reading habit and promote their development. In particular, young children are expected to actively read picture books with an art style they like, and reading picture books with a variety of content is expected to promote their development.

[0003] A common method for searching for picture books is to use a general-purpose web search engine such as Google® (see, for example, Non-Patent Document 1). One possible method for obtaining picture books with a preferred art style from the target audience is to devise text that describes the target audience's preferred art style (for example, "anime-style picture books"), provide this text as a query to the web search engine, and obtain the search results.

[0004] A second method for obtaining picture books in the target audience's preferred art style as search results is to provide images in the target audience's preferred art style as a query to a web search engine and obtain the search results.

[0005] Google, [online], [Searched on February 17, 2025], Internet <URL: https: / / www.google.co.jp / >

[0006] If the artistic style can be appropriately described in text, the first method can be used to obtain picture books with the artistic style preferred by the target audience as search results. However, for young children and other target audiences, it is not easy to appropriately describe the artistic style in text. Therefore, the first method has the challenge of making it difficult to obtain picture books with the artistic style that the target audience truly prefers as search results.

[0007] If there are images of the subject's preferred art style, the second method can be used to obtain picture books in that style as search results. However, this requires, for example, showing the subject, such as a young child, many images of picture book covers and having them select the image of their preferred art style. However, when showing many images of picture book covers to a young child, they may make their selection based on factors such as the perceived interest of the story from the cover, and the image of their preferred art style may not be selected. Therefore, the second method has the problem that it may result in obtaining picture books in an art style that the subject does not prefer.

[0008] The disclosure technology aims to provide a method for obtaining picture books with an art style preferred by target audiences such as young children as search results.

[0009] An image acquisition method according to one aspect of the disclosed technology includes: a display step in which a display unit presents a presentation image in which presentation book images of the same motif but with different drawing styles are arranged in different positions; an input receiving step in which an input receiving unit receives a selection input operation to select a presentation book image with a drawing style preferred by the subject from among a plurality of presentation book images included in the presentation image; and an image selection control step in which an image selection control unit obtains a presentation book image corresponding to the selection input operation received by the input receiving unit from among a plurality of presentation book images included in the presentation image as a search query image.

[0010] According to the disclosed technology, it is possible to provide a technology that allows users to obtain picture books with an art style preferred by target audiences such as young children as search results.

[0011] Figure 1 shows an example of the functional configuration of an image acquisition device. Figure 2 shows an example of the processing of an image acquisition method. Figure 3 shows an example of the processing of the image selection control unit 4. Figure 4 shows an example of the display of the user interface. Figure 5 shows an example of the display of the user interface. Figure 6 shows an example of the display of the user interface. Figure 7 shows an example of the display of the user interface. Figure 8 shows an example of the display of the user interface. Figure 9 shows an example of the display of the user interface. Figure 10 shows an example of the display of the user interface. Figure 11 shows an example of the functional configuration of a computer.

[0012] Embodiments of the disclosed technology will be described below with reference to the drawings. Note that components having the same function are numbered identically in the drawings, and redundant explanations are omitted.

[0013] [Image Acquisition Device and Method] As shown in Figure 1, the image acquisition device 10 includes, for example, a book database 1, a presentation material acquisition unit 2, a presentation image acquisition unit 3, an image selection control unit 4, a display unit 5, and an input reception unit 6. As illustrated in Figure 1, the image acquisition device 10 may be included in the search device 20. The search device 20 includes, for example, the image acquisition device 10, a search book database 7, a search unit 8, and a search result image generation unit 9.

[0014] The image acquisition method is implemented, for example, by having each component of the image acquisition device 10 perform the processing shown in steps S2 to S4 in Figure 2. When the search device 20, which includes the image acquisition device 10, performs search processing, the processing shown in steps S8 and S9, indicated by dashed lines in Figure 2, is further performed.

[0015] The image acquisition device 10 does not necessarily have to include at least one of the following: book DB 1, presentation material acquisition unit 2, presentation image acquisition unit 3, image selection control unit 4, display unit 5, and input reception unit 6.

[0016] For example, the image acquisition device 10 does not necessarily have to include a presentation image acquisition unit 3, an image selection control unit 4, a display unit 5, and an input receiving unit 6. That is, for example, the image acquisition device 10 may only include a book DB 1 and a presentation material acquisition unit 2. In this case, the process of step S2 in Figure 2 is performed. In this case, the image acquisition device 10 becomes a device that obtains N presentation book images to be presented in order to allow the user to select one or more images from among the N presentation book images, up to a number less than N. N is an integer of 2 or more. N may also be an integer of 3 or more and up to 9. By setting N to 3 or more, the range of choices for the user is broadened. The image acquisition device 10 may also be a device that obtains N presentation book images to be presented in order to allow the user to select one image from among the N presentation book images.

[0017] For example, the image acquisition device 10 does not have to include a book DB 1, a presentation material acquisition unit 2, an image selection control unit 4, a display unit 5, and an input receiving unit 6. That is, for example, the image acquisition device 10 may only include a presentation image acquisition unit 3. In this case, the process of step S3 in Figure 2 is performed. In this case, the image acquisition device 10 is a device that obtains a presentation image containing N presentation book images, which is an image to be presented to the user in order to select one or more but less than N images from N presentation book images with the same motif but different drawing styles. The image acquisition device 10 may also be a device that obtains a presentation image containing N presentation book images, which is an image to be presented to the user in order to select one image from N presentation book images with the same motif but different drawing styles.

[0018] Furthermore, the image acquisition device 10 does not necessarily have to include a book database 1, a presentation material acquisition unit 2, and a presentation image acquisition unit 3. That is, for example, the image acquisition device 10 may only include an image selection control unit 4, a display unit 5, and an input reception unit 6. In this case, the process of step S4 in Figure 2 is performed. In this case, the image acquisition device 10 becomes a device that obtains, as a search query image, a presentation book image corresponding to information indicating which of the multiple presentation book images contained in each S (S is an integer of 1 or more) presentation image was selected.

[0019] The example of the target group is an infant. The target group can be a child or an adult.

[0020] The following describes each component of the image acquisition device 10.

[0021] <Book DB1> Book DB1 is a database that pre-stores images for at least one page contained in each of multiple books. If the presentation material acquisition unit 2 uses only images, it is sufficient that Book DB1 stores images for at least one page contained in each of multiple books. If the presentation material acquisition unit 2 uses both text and images, it is sufficient that Book DB1 stores the text and images contained in each of the pages contained in each of the multiple books, with the text and images contained in those pages associated with each page. Hereafter, the text and images contained in a page may be referred to as "information contained in the page".

[0022] A book is a publication containing both text and illustrations, and may be a printed book such as a book or magazine, or an electronic book. An example of a book is a picture book. An example of an electronic book is an electronic document written and structured using a markup language such as XML.

[0023] In the following explanation, we may use the example of a picture book.

[0024] The book database 1 only needs to contain information on a large number of existing books, and at a minimum, it should contain information on more books than the number of book images for presentation obtained by the presentation material acquisition unit 2, which will be described later.

[0025] "One or more pages included in the book" means, for example, all the pages included in the book. However, it is not necessary for "one or more pages included in the book" to be all the pages included in the book; for example, it could be several main pages, one representative page, or the cover page, which is an example of one representative page.

[0026] "Images included on a page" refers to, for example, the image of the page itself, or the image of the entire page. However, "images included on a page" may also refer to images of only a part of the page, such as an image of the main part of the page.

[0027] <Presentation Material Acquisition Unit 2> The presentation material acquisition unit 2 obtains images of one page from each of the N books in the book DB 1 as N presentation book images. The presentation material acquisition unit 2 outputs the N presentation book images to the presentation image acquisition unit 3 (step S2).

[0028] The presentation material acquisition unit 2 performs a process to obtain N presentation book images that have the same motif but different drawing styles. This process includes prioritizing the selection of pages from each book stored in the book DB 1 that have high similarity in the text contained on the page and / or similarity in the group of objects that make up the picture on the page.

[0029] A motif is at least one of the subject, a central symbolic element of the expression, or a style of expression. Examples of subjects include famous stories such as Little Red Riding Hood and Cinderella. Examples of central symbolic elements of the expression include subjects of depiction such as trains and cars. Examples of styles of expression include forms with visual and structural characteristics, such as illustrated scrolls.

[0030] Furthermore, books and images sharing the same motif can be classified into the same category.

[0031] Drawing style refers to the artistic aspects of an image, such as the method of representing objects and the use of color, and does not include composition. Composition, on the other hand, refers to the physical aspects of an image, such as the arrangement of objects and their relative positions. Drawing style can also be expressed as art style or brushwork. Since drawing style does not include composition, it can be expressed as "drawing style (excluding composition)" or "drawing style excluding composition." Similarly, since art style does not include composition, it can be expressed as "art style (excluding composition)" or "art style excluding composition."

[0032] For example, the presentation material acquisition unit 2 may perform at least any one of the following first to fifth methods. Further, the presentation material acquisition unit 2 may perform the following sixth method in addition to at least any one of the following first to fifth methods.

[0033] <<First Method>> The first method is an example in which the presentation material acquisition unit 2 performs processing including, as processing for obtaining N presentation book images having the same motif but different artistic styles, preferentially selecting pages included in each book stored in the book database 1 that have high similarity of text contained in the pages.

[0034] The book DB 1 stores, for each of a plurality of books, image of pictures and text of each page in association with each other.

[0035] The presentation material acquisition unit 2 performs the following steps 11 to 12.

[0036] Step 11: First, the presentation material acquisition unit 2 acquires text of one page of one book stored in the book DB 1 as similarity evaluation target text. Further, the presentation material acquisition unit 2 may acquire the image of the page as a similarity evaluation result image.

[0037] For example, assume that in one picture book of Little Red Riding Hood, the fifth page is a scene where Little Red Riding Hood is walking through the forest toward her grandmother's house. The presentation material acquisition unit 2 takes the text of the fifth page as the similarity evaluation target text, and acquires the image of the fifth page as a first similarity evaluation result image.

[0038] Step 12: Next, the presentation material acquisition unit 2 receives the similarity evaluation target text obtained in step 11 as input, performs similarity search on page-unit book text in the book DB 1, and acquires images of pages corresponding to search results with higher similarity degrees as similarity evaluation result images.

[0039] Here, as the degree of similarity between two texts, a well-known similarity may be used. For example, a value representing the degree of commonality of word sets appearing in two texts expressed by the Jaccard coefficient may be used; alternatively, for each text, a text-derived vector may be obtained using generative AI provided in another device, and the cosine similarity or Euclidean distance between the vectors derived from the two texts may be used. Of course, as the degree of similarity between two texts, vector similarity based on TFIDF values (for example, cosine similarity or Euclidean distance) may also be used.

[0040] In the aforementioned example of Little Red Riding Hood, although there are many picture books created by various authors, almost all of these picture books follow the same plot development that Little Red Riding Hood walks in the forest and meets the wolf. Therefore, even if step 12 is performed without limiting to the Little Red Riding Hood picture books in the book DB 1, the images of the pages corresponding to the top search results will still be images of the scene where Little Red Riding Hood is walking through the forest towards Grandma's house.

[0041] Through the above step 11 and step 12, a plurality of similarity evaluation result images can be obtained. In the aforementioned example of Little Red Riding Hood, in step 12, a similarity evaluation target text, which is a text corresponding to a similarity evaluation result image that is an image of a certain page of a certain Little Red Riding Hood picture book, is used as input, and images that are included in other books and show a scene similar to the similarity evaluation target text are obtained as similarity evaluation result images.

[0042] The presentation material acquisition unit 2 may use all of the plurality of similarity evaluation result images obtained through step 11 and step 12 as the plurality of presentation book images; alternatively, the plurality of similarity evaluation result images obtained through step 11 and step 12 may be used as intermediate candidates, and a final result selected from the intermediate candidates by the sixth method described later may be obtained as the plurality of presentation book images.

[0043] If the number of book images to be presented is N (where N is an integer greater than or equal to 2), then if all of the similarity evaluation result images are to be used as book images to be presented, then in the similarity search in step 12, it is sufficient to obtain the top N-1 images with the greatest similarity as the similarity evaluation result images. In this case, if no similarity evaluation result images are obtained in step 11, then in the similarity search in step 12, the top N images with the greatest similarity are obtained as N similarity evaluation result images, and these N similarity evaluation result images can be used as N book images to be presented.

[0044] If multiple similarity evaluation result images are used as intermediate candidates, and the final result selected from these intermediate candidates by the sixth method described later is to be N book images for presentation, then in the similarity search in step 12, the top K images (K is an integer greater than N) with the greatest similarity are obtained as K similarity evaluation result images, and in the selection process described later, N images from the K similarity evaluation result images are selected as N book images for presentation. As mentioned above, K can be any integer greater than N, so in the similarity search in step 12, if the number of images with a similarity of greater than or equal to the threshold exceeds N, the K similarity evaluation result images may be those with a similarity of greater than or equal to the threshold.

[0045] Furthermore, while the multiple similarity evaluation result images obtained by the first method may include images of books with multiple titles, or images of multiple scenes from a single title, the multiple presentation book images, which may be all or part of the multiple similarity evaluation result images, only need to have similar composition and content, and different artistic styles. Therefore, no operational problems arise. The same applies to the second to fourth methods described later.

[0046] Thus, the presentation material acquisition unit 2 may obtain N presentation book images by a process that includes selecting images of a predetermined number of pages with the highest degree of similarity between the text on each page and the text to be evaluated for similarity, from among the pages contained in each book other than the first book stored in the book DB1. The predetermined number is N or N-1.

[0047] <<Second Method>> Similar to the first method, the second method is one of the methods in which the presentation material acquisition unit 2 performs the process of obtaining N presentation book images, which are the same motif but have different artistic styles. This process includes prioritizing the selection of pages with high similarity in the text contained on them from among the pages contained in each book stored in the book DB1.

[0048] Book DB1 stores images and text from each page of multiple books, with their corresponding images.

[0049] The presentation material acquisition unit 2 performs steps 21 to 23 below, with M being one value greater than or equal to N.

[0050] Step 21: The presentation material acquisition unit 2 first obtains the text of a page from a book stored in the book DB 1 as the first similarity evaluation target text. The presentation material acquisition unit 2 also obtains an image of the said page as the similarity evaluation result image.

[0051] Step 22: Following Step 21, the presentation material acquisition unit 2 selects the image of the page containing the largest similarity between the first similarity evaluation target text and the m-1 similarity evaluation target text (for example, the sum or average value of the similarity between the first similarity evaluation target text and the m-1 similarity evaluation target text) from each book stored in the book DB1 that is not one of the first books, with m being an integer between 2 and M-1, in order, and processes the text of that page to be the m-th similarity evaluation target text.

[0052] Step 23: Following Step 22, the presentation material acquisition unit 2 selects the image of the page containing the text that is not one of the first book to the M-1 book stored in the book DB1, and which has the greatest degree of similarity between the first similarity evaluation target text and the M-1 similarity evaluation target text (for example, the sum or average value of the similarity between the first similarity evaluation target text and the M-1 similarity evaluation target text).

[0053] Steps 21 to 23 above yield M similarity evaluation result images.

[0054] The presentation material acquisition unit 2 may use all M=N similarity evaluation result images obtained by steps 21 to 23 with M=N as N presentation book images, or it may use K similarity evaluation result images obtained by steps 21 to 23 with M=K>N as intermediate candidates, and obtain the final result selected from the intermediate candidates by the sixth method described later as N presentation book images.

[0055] <<Third Method>> In the first and second methods, since the same scene in picture books with the same title has similar text, the evaluation is based on the degree of similarity to the text on a given page. However, since the same scene in picture books with the same title not only has similar text but also features similar compositions and sets of objects, the evaluation can also be based on information about the composition and objects in the image on a given page. This is the third method.

[0056] The third method is an example in which the presentation material acquisition unit 2 performs the process of obtaining N presentation book images, which are the same motif but have different artistic styles. This process includes prioritizing the selection of pages from each book stored in the book DB1 that have a high degree of similarity in the group of objects that make up the illustrations on the pages.

[0057] Book DB1 stores images of the illustrations on each page for each of several books.

[0058] The presentation material acquisition unit 2 performs the following steps 31 to 34.

[0059] Step 31: The presentation material acquisition unit 2 first obtains an image of a page from a book (the first book) stored in the book DB 1 as the image to be evaluated for similarity. The presentation material acquisition unit 2 may also obtain this image to be evaluated for similarity as the image of the similarity evaluation result.

[0060] Step 32: The presentation material acquisition unit 2 then performs object recognition on the image to be evaluated for similarity obtained in step 31, thereby obtaining a set of objects contained in the image as a set of objects to be evaluated for similarity.

[0061] Step 33: The presentation material acquisition unit 2 also uses the images of pages contained in each of the books other than the first book stored in the book DB1 as candidate images, and performs object recognition on each candidate image to obtain a set of objects contained in each candidate image as a candidate image object set.

[0062] Step 34: After steps 32 and 33, the presentation material acquisition unit 2 obtains images of pages from each of the books other than the first book stored in the book DB1, where the candidate image object set of the page has a higher degree of similarity between the set and the object set to be evaluated for similarity, as similarity evaluation result images. As an index to represent the degree of similarity between sets, for example, a well-known inter-set similarity index such as the Jaccard coefficient may be used.

[0063] Multiple similarity evaluation result images are obtained through steps 31 to 34 described above. The actions performed by the presentation material acquisition unit 2 after obtaining the multiple similarity evaluation result images are the same as in the first method, so a detailed explanation is omitted here. The presentation material acquisition unit 2 may use all of the multiple similarity evaluation result images obtained through steps 31 to 34 as multiple presentation book images, or it may use the multiple similarity evaluation result images obtained through steps 31 to 34 as intermediate candidates and obtain the final result selected from the intermediate candidates by the sixth method described later as multiple presentation book images.

[0064] Furthermore, the presentation material acquisition unit 2 may obtain a similarity evaluation result image by performing an evaluation that combines the similarity evaluation in step 12 of the first method and the similarity evaluation in step 34 of the third method.

[0065] Thus, the presentation material acquisition unit 2 may obtain N presentation book images by performing object recognition on an image of a page contained in the first book stored in the book DB 1, thereby obtaining a set of objects contained in the image as a similarity evaluation target object set, and by performing object recognition on each candidate image of a page contained in each book other than the first book stored in the book DB 1, thereby obtaining a set of objects contained in each candidate image as a candidate image object set, and by selecting a predetermined number of page images with the highest degree of similarity between the set of candidate image object set and the similarity evaluation target object set from among the pages contained in each book other than the first book stored in the book DB 1. The predetermined number is N or N-1.

[0066] <<Fourth Method>> Similar to the third method, the fourth method is one of the methods in which the presentation material acquisition unit 2 performs the process of obtaining N presentation book images, which are the same motif but have different drawing styles. This process includes prioritizing the selection of pages from each book stored in the book DB1 that have a high degree of similarity in the group of objects that make up the illustrations on the pages.

[0067] Book DB1 stores images of the illustrations on each page for each of several books.

[0068] The presentation material acquisition unit 2 performs steps 41 to 45 below, with M being one value greater than or equal to N.

[0069] Step 41: The presentation material acquisition unit 2 first obtains an image of a page from a book (first book) stored in the book DB 1 as the first similarity evaluation target image. The presentation material acquisition unit 2 may also obtain the first similarity evaluation target image as the similarity evaluation result image.

[0070] Step 42: After step 41, the presentation material acquisition unit 2 performs object recognition on the first similarity evaluation target image to obtain a set of objects included in the first similarity evaluation target image as the first similarity evaluation target object set. The presentation material acquisition unit 2 may also obtain the image on this page as the similarity evaluation result image.

[0071] Step 43: The presentation material acquisition unit 2 uses the images of pages contained in each book other than the first book stored in the book DB1 as candidate images, and performs object recognition on each candidate image to obtain a set of objects contained in each candidate image as a candidate image object set.

[0072] Step 44: After performing steps 42 and 43, the presentation material acquisition unit 2 selects the image of the page with the largest sum or average value of inter-set similarity between the first similarity evaluation target object set and the m-1 similarity evaluation target object set of the candidate page image object sets contained in each book that is not one of the first book to the m-1 book stored in the book DB1, where m is an integer from 2 to M-1 in order, and processes the image of that page to be set as the m-th similarity evaluation target object set.

[0073] Step 45: Following Step 44, the presentation material acquisition unit 2 selects the image of the page from among the pages in the books stored in the book DB1 that are not among the first book to the M-1 book, the page image that has the largest sum or average value of inter-set similarity between the first similarity evaluation target object set and the M-1 similarity evaluation target object set of the candidate page image object set.

[0074] Steps 41 to 45 above yield M similarity evaluation result images. For inter-set similarity, for example, the well-known Jaccard coefficient can be used.

[0075] The presentation material acquisition unit 2 may use all M=N similarity evaluation result images obtained by steps 41 to 45 with M=N as N presentation book images, or it may use K similarity evaluation result images obtained by steps 41 to 45 with M=K>N as intermediate candidates, and obtain the final result selected from the intermediate candidates by the sixth method described later as N presentation book images.

[0076] The presentation material acquisition unit 2 may obtain a similarity evaluation result image by performing an evaluation that combines the similarity evaluation in step 22 of the second method and the similarity evaluation in step 44 of the fourth method, and an evaluation that combines the similarity evaluation in step 23 of the second method and the similarity evaluation in step 45 of the fourth method.

[0077] <<Fifth Method>> For well-known titles like Little Red Riding Hood, since there are picture books written by multiple authors, it is acceptable to use the cover images of books with the same title as the similarity evaluation result images. This is the fifth method.

[0078] When the presentation material acquisition unit 2 performs the fifth method, the book DB 1 stores, for each of the multiple books, the title, unique information and / or author information and / or version information, and a cover image.

[0079] Here, the author may be the author of the book, or the author of the illustrations included in the book.

[0080] The presentation material acquisition unit 2 obtains images of the covers of multiple books from the book DB 1 as similarity evaluation result images, each having the same title but differing in at least one of the unique information, author information, or version information. The presentation material acquisition unit 2 may use all of the obtained multiple (M=N) similarity evaluation result images as multiple (N) presentation book images, or it may use the obtained multiple (K, however M=K>N) similarity evaluation result images as intermediate candidates and obtain the final result selected from the intermediate candidates by the sixth method described later as multiple (N) presentation book images.

[0081] Thus, the presentation material acquisition unit 2 may perform a process that involves acquiring cover images from each of the N books in the book DB 1 to create N presentation book images, which may include selecting cover images from the book DB for M books (where M is an integer greater than or equal to N) that have the same title but differ in at least one of the unique information, author information, or version information.

[0082] <<Sixth Method>> In the case of well-known titles such as Little Red Riding Hood, there may be more than a dozen picture books with the same title. In this case, there will be more than a dozen images that have a similarity score greater than or equal to the threshold in the similarity search. If all of these more than a dozen images are included as presentation images for the book to be shown to the subject, it may become difficult for the subject to make a selection due to reasons such as having too many options or including images with similar art styles.

[0083] In such cases, as briefly mentioned in the explanation of the first method, the K similarity evaluation result images obtained by the first to fifth methods may be used as intermediate candidate book images for presentation, and N of these K intermediate candidate book images, whose similarity based on image features is as different as possible, may be selected as the final book images for presentation. This is the sixth method.

[0084] As image features and similarity based on image features, those known from Japanese Patent Publication No. 2017-129901, etc., can be used.

[0085] For example, if the image features are a color histogram, which is a group of image features that represent hue, then similarity measures such as the Bhattacharyya coefficient or the reciprocal of difference measures such as the Hellinger distance or Kullback-Leibler divergence can be used to measure their similarity.

[0086] In other words, in the sixth method which uses at least one of the first to fourth methods, the presentation material acquisition unit 2 should first perform step 61 below, and then perform step 62 below.

[0087] Step 61: The presentation material acquisition unit 2 selects pages from each of the books stored in the book DB1 that have high similarity in the text contained on the page and / or similarity in the group of objects that make up the picture on the page, thereby obtaining images of one page from each of the K books as K intermediate candidate book images for presentation.

[0088] Step 62: The presentation material acquisition unit 2 obtains N presentation book images by selecting N images from K intermediate candidate book images for presentation that have large differences in similarity based on image features.

[0089] Step 62 can be implemented, for example, by obtaining N book images for presentation that represent the combination with the largest sum of differences in the paired image features included. Specifically, the presentation material acquisition unit 2 should perform steps 62A1 to 62A2 below as step 62.

[0090] Step 62A1: The presentation material acquisition unit 2 acquires a set of all N presentation intermediate candidate book images contained in the K presentation intermediate candidate book images ( Ｋ C Ｎ For each N-group, obtain the sum or average of the similarity values ​​based on the image features of all pairs included in each N-group.

[0091] Step 62A2: The material acquisition unit 2 for presentation is Ｋ C ＮThe images included in the N-group of N images that has the smallest sum or mean similarity obtained in step 62A2 are designated as the N book images to be presented.

[0092] Step 62 may be implemented, for example, by sequentially selecting the image whose image features differ most from those of the already selected image. Specifically, the presentation material acquisition unit 2 can perform steps 62B1 to 62B2 below as step 62.

[0093] Step 62B1: The presentation material acquisition unit 2 obtains one of the K intermediate candidate book images for presentation as the first presentation book image.

[0094] Step 62B2: The presentation material acquisition unit 2 selects the intermediate candidate presentation book image that is not any of the first presentation book image to the (n-th) presentation book image, where n is an integer between 2 and N, in order, and the sum or average value of the similarity between the first presentation book image and each of the (n-th) presentation book images based on image features is smallest, as the nth presentation book image.

[0095] Furthermore, in the sixth method using the fifth method, the presentation material acquisition unit 2 should first perform the following step 61' and then perform the above step 62.

[0096] Step 61': The presentation material acquisition unit 2 selects from the book DB1 books that have the same title but differ in at least one of the unique information, author information, and version information, thereby obtaining images of one page from each of the K books (where K is an integer greater than N) as K intermediate candidate book images for presentation.

[0097] At the beginning of the explanation of the sixth method, we described the case where the number of images K whose similarity in the similarity search is greater than or equal to the threshold is greater than the number of presentation book images N. However, it is not mandatory to perform threshold comparison in the similarity search. Alternatively, the top K images with the highest similarity in the similarity search may be used as intermediate candidate book images for presentation, or K intermediate candidate book images for presentation may be obtained by some method, including manual work. Therefore, the presentation material acquisition unit 2 may take the K intermediate candidate book images for presentation prepared by some method as input and perform only step 62, or only steps 62A1 and 62A2 which are concrete examples of step 62, or only steps 62B1 and 62B2 which are concrete examples of step 62. In this case, the K "intermediate candidate book images for presentation" may not necessarily be "intermediate candidates," so they may be called "candidate book images for presentation" instead of "intermediate candidate book images for presentation."

[0098] <<Method for obtaining multiple sets of N book images for presentation>> The presentation material acquisition unit 2 may obtain multiple sets of N book images for presentation. Hereafter, a set of N book images for presentation will also be called a book image set for presentation. That is, the presentation material acquisition unit 2 obtains S sets (where S is an integer of 1 or more) of book image sets for presentation.

[0099] When the presentation material acquisition unit 2 obtains multiple sets of presentation book image sets, it obtains multiple sets of presentation book image sets such that the motifs differ among the multiple sets of presentation book image sets. Multiple sets of presentation book image sets with different motifs can be obtained, for example, by using different similarity evaluation target texts in a first method performed multiple times, using different first similarity evaluation target texts in a second method performed multiple times, using different similarity evaluation target images in a third method performed multiple times, using different first similarity evaluation target images in a fourth method performed multiple times, and using different titles in a fifth method performed multiple times.

[0100] <Image Acquisition Unit 3 for Presentation> The Image Acquisition Unit 3 for Presentation receives S sets of book images for presentation (S is an integer of 1 or more). The S sets of book images for presentation input to the Image Acquisition Unit 3 may be obtained by the Material Acquisition Unit 2 or by other means. However, each set of book images for presentation consists of N book images with the same motif but different artistic styles, and if S is 2 or more, the motifs are different in the S sets of book images for presentation.

[0101] The presentation image acquisition unit 3 obtains S presentation images by acquiring one presentation image for each set of presentation book images (step S3). The presentation image acquisition unit 3 outputs the S presentation images to the image selection control unit 4.

[0102] Examples of three presentation images when S = 3 are the presentation images included in the displays from Figure 8 to Figure 10. The motif in the presentation image included in Figure 8 is a "bear," the motif in the presentation image included in Figure 9 is a "face," and the motif in the presentation image included in Figure 10 is "three little pigs." As in this example, the motifs in multiple presentation images are different from each other.

[0103] <<Method for obtaining a single presentation image>> The presentation image acquisition unit 3 obtains a presentation image by arranging N presentation book images included in the presentation book image set at different positions within the background image. That is, the presentation image acquisition unit 3 obtains a presentation image in which N presentation book images included in the presentation book image set are arranged at different positions. The background image can be any image, but it is preferable that it is an image that does not contain objects or color schemes related to the presentation book images, such as an image based on a single color.

[0104] For example, if there are a predetermined N positions within the area of ​​a background image that is the same size as the image to be presented, and each of the predetermined N positions is a predetermined nth position (where n is any integer from 1 to N), then the image acquisition unit 3 can obtain the image to be presented by placing any nth image from the set of images to be presented at the predetermined nth position within the background image for each n.

[0105] Furthermore, it is not mandatory for the images of the books to be displayed to be placed in a specific location within the display image; as mentioned above, the images of the books to be displayed can be placed in different locations within the display image.

[0106] <<First method for obtaining multiple presentation images>> When the presentation image acquisition unit 3 obtains multiple presentation images (i.e., when S is an integer of 2 or more), the presentation image acquisition unit 3 can obtain multiple (S) presentation images by performing the "method for obtaining one presentation image" described above for each set of presentation book images to obtain one presentation image, and doing this multiple times (S times).

[0107] Note that the number of presentation book images included in multiple (S) presentation images does not need to be the same. That is, if s is an integer between 1 and S, and the s-th presentation image contains Ns presentation book images, then N １ ~N Ｓ The values ​​do not all need to be the same, N １ ~N Ｓ Some or all of the values ​​may be different. In this case, in each process of obtaining a set of presentation book images, which is performed S times by the presentation material acquisition unit 2, Ns presentation book images should be obtained in the sth process. Note that Ns is an integer of 2 or more, just like N. Ns may be an integer of 3 or more, or even an integer of 9 or less.

[0108] However, as will be explained later, multiple presentation images are displayed sequentially on the screen, and the user is asked to select the presentation book image with their preferred art style for each display. Therefore, from the perspective of simplifying the user's selection process as much as possible, and from the perspective of the implementation that generates the presentation images, it is preferable that the number of presentation book images displayed sequentially be the same.

[0109] <<Second Method for Obtaining Multiple Presentation Images>> As described later, the purpose of obtaining multiple presentation images is to allow the subject to repeatedly identify their preferred art style by sequentially displaying them on the screen and having them select the presentation book image with their preferred art style for each display. When obtaining multiple presentation images using the first method, there is a possibility that presentation book images with similar art styles, although with different motifs, may be placed in the same position among the multiple presentation images. Due to positional bias, the presentation book image with the most preferred art style may not be selected.

[0110] To avoid this, the goal is to ensure that, in multiple displays, images of books with an undesirable art style are placed at the viewer's gaze point. However, it is impossible to know the viewer's gaze point or preferred art style in advance. Therefore, in multiple displays, i.e., in multiple presentation images, it is best to place images of books with as different image features as possible at the same position. The second method achieves this.

[0111] In the second method, if there are a predetermined N positions within the area of ​​a background image that is the same size as the presentation image, and each of the predetermined N positions is a predetermined nth position (where n is any integer from 1 to N), the presentation image acquisition unit 3, when performing the process of obtaining S presentation images (here, S is an integer of 2 or more) by arranging the N presentation book images included in each presentation book image set at the predetermined N positions within each presentation image, arranges the N presentation book images included in each presentation book image set within each presentation image in such a way that for all n, the degree of similarity based on the image features of the S presentation book images placed at the predetermined nth position in the S presentation images is reduced.

[0112] Furthermore, depending on the operation of the presentation material acquisition unit 2 and the presentation image acquisition unit 3, there is a very high possibility that multiple presentation images will be placed in the same position, each with a different motif but a similar style. For example, if the presentation material acquisition unit 2 obtains a similarity evaluation result image (i.e., the first similarity evaluation result image) in one of steps 11, 21, 31, or 41 and uses it as the first presentation book image, and the presentation image acquisition unit 3 places the first presentation book image in the first position, then the similarity evaluation result image obtained by the presentation material acquisition unit 2 is an image of a page from a particular book stored in the book DB1, which the presentation material acquisition unit 2 has selected using some algorithm. Furthermore, if this algorithm pre-defines a reference image feature, which is a set of reference image features, and selects an image from the pages stored in the book DB1 that is as close as possible to the reference image feature, then the style of the first presentation book image in multiple sets of presentation book images will be similar, and as a result, the style of the presentation book image placed in the first position in multiple presentation images will be similar.

[0113] To avoid this, the presentation image acquisition unit 3 may, as a second method, obtain S presentation images by randomly arranging N presentation images, each of the S presentation book image sets, at predetermined N positions, rather than by placing the nth presentation book image included in each of the S presentation book image sets at predetermined N positions.

[0114] <<Third Method for Obtaining Multiple Presentation Images>> Multiple candidate presentation images may be obtained by the first method or the second method for obtaining multiple presentation images, and one or more of the candidate presentation images that are suitable for the subject's attributes may be selected as presentation images. This is the third method. Subject attributes include information that describes the characteristics of the area in which the subject lives, such as whether the subject lives in an area near the sea or an area near the mountains, and also information that describes the subject's interests, such as whether the subject is interested in the sea or an interest in the mountains. Note that subject attributes may also include at least one of the following: the subject's age, gender, favorite color, favorite genre of books.

[0115] In the third method, with Q being an integer greater than or equal to 1 and R being an integer greater than Q, the presentation image acquisition unit 3 first performs steps 101 and 102 below, and then, with steps 101 and 102 already completed, performs steps 103 and 104 below.

[0116] Step 101: The presentation image acquisition unit 3 first obtains R presentation candidate images by performing the "first method for obtaining multiple presentation images" or the "second method for obtaining multiple presentation images," replacing "S" with "R" and "presentation image" with "presentation candidate image" in the above description. Similarly, a "presentation book image" included in the presentation candidate images may also be called a "presentation candidate book image."

[0117] Step 102: Following Step 101, the presentation image acquisition unit 3 associates information identifying the motif and / or information representing the degree of suitability to the attributes with each of the R candidate presentation images. The information identifying the motif and / or information representing the degree of suitability to the attributes may be provided in advance for each book stored in the book DB1 and read by the presentation image acquisition unit 3 from the book DB1, or the presentation image acquisition unit 3 may accept input of information obtained by some method, such as manual work, for each of the R candidate presentation images. If information identifying the motif is associated with each of the R candidate presentation images, information representing the degree of suitability to each attribute for each motif is also associated with it.

[0118] Step 103: With steps 101 and 102 already completed, the presentation image acquisition unit 3 obtains information representing the subject's attributes from the input reception unit 6, which will be described later. If the attribute is age, the subject's attribute may be an age within the range of the age entered into the input reception unit 6 minus a predetermined integer C1, and within the range of the age entered into the input reception unit 6 plus a predetermined integer C2. C1 and C2 are positive integers of 1 or greater, for example, 1.

[0119] Step 104: Following step 103, the presentation image acquisition unit 3 obtains Q presentation images by prioritizing the selection of presentation candidate images from among the R presentation candidate images that have a degree of suitability to the attributes that is close to the attributes of the subject obtained in step 103. If, in step 102, information identifying a motif is associated with each of the R presentation candidate images, the presentation image acquisition unit 3 should obtain Q presentation images by prioritizing the selection of presentation candidate images of motifs that have a degree of suitability to the attributes that is close to the attributes of the subject obtained in step 103.

[0120] In the third method, the Q presentation images obtained in step 104 of the presentation image acquisition unit 3 become the S presentation images output to the image selection control unit 4.

[0121] <<Fourth Method for Obtaining Multiple Presentation Images>> The third method obtains presentation images that are suitable for the subject's attributes, but there may be cases where information about the subject's attributes cannot be obtained or is not appropriate to obtain. Therefore, a method may be adopted that increases the likelihood that one of the multiple presentation images obtained will be suitable for the subject's attributes, regardless of the subject's attributes. An example of this method will be explained as the fourth method, where Q is an integer of 2 or more and R is an integer greater than Q.

[0122] If the presentation image acquisition unit 3 randomly selects Q presentation images from R candidate presentation images, Q images with similar motifs will be selected as presentation images. This may result in a selection of many presentation images suitable for one attribute, but fewer suitable for another attribute. Therefore, in the fourth method, the presentation image acquisition unit 3 selects Q presentation images from the R candidate presentation images so that their motifs are as different as possible, with the aim of increasing the probability that at least one presentation image suitable for each attribute is included among the Q presentation images. Specifically, in the fourth method, the presentation image acquisition unit 3 performs the following steps 111 to 114.

[0123] Step 111: The presentation image acquisition unit 3 first performs either the "first method for obtaining multiple presentation images" or the "second method for obtaining multiple presentation images," replacing "S" with "R" and "presentation image" with "candidate presentation image" in the above description, thereby obtaining R candidate presentation images. In step 112, the "presentation book image" included in the candidate presentation images is referred to as the "candidate presentation book image."

[0124] Step 112: The presentation image acquisition unit 3 obtains the text contained in each candidate presentation image as the representative text of the candidate presentation image.

[0125] As can be seen from the explanation of the presentation material acquisition unit 2 in example 1 of step 112, the text is almost the same in the multiple presentation candidate book images included in each presentation candidate image. Therefore, the presentation image acquisition unit 3 either obtains the text contained in any one of the presentation candidate book images included in each presentation candidate image using well-known techniques and uses it as the representative text, or it inputs any of the similarity evaluation target texts obtained by the presentation material acquisition unit 2 into the presentation image acquisition unit 3 and obtains the input similarity evaluation target text as the representative text. Note that the presentation image acquisition unit 3 may use a set of morphemes contained in the aforementioned presentation book images or similarity evaluation target text as the representative text, or a set of morphemes consisting only of words that have meaning on their own (so-called content words or independent words) from the morphemes contained in the text (i.e., a set obtained by removing particles, conjunctions, etc. from the set of morphemes contained in the text, leaving only words that have meaning on their own). The set of morphemes can be obtained using well-known morphological analysis techniques, etc.

[0126] Step 112 Example 2: For each candidate image to be presented, the image acquisition unit 3 obtains a representative text by concatenating the text contained in multiple (two or more, or all) candidate book images to be presented that are included in the candidate image to be presented. The text contained in each candidate book image to be presented may be obtained using well-known techniques, or it may be obtained by the image acquisition unit 2 to be used. Note that the image acquisition unit 3 may use not the concatenated text itself, but rather a set of morphemes contained in the concatenated text, or a set of morphemes consisting only of words that have meaning on their own from among the morphemes contained in the text, as the representative text.

[0127] Example 3 of step 112: For each candidate presentation image, the presentation image acquisition unit 3 uses, as the representative text, either a set of morphemes that commonly appear in the text included in a plurality of (two or more, or all) candidate book images for presentation included in said candidate presentation image, or a set of morphemes consisting solely of words that have independent meaning from among the morphemes that commonly appear in the text included in the plurality of candidate book images for presentation included in said candidate presentation image. Specifically, for example, for each candidate presentation image, the presentation image acquisition unit 3 obtains the text included in each candidate presentation book image included in said candidate presentation image in the same manner as in Example 2, obtains from the text included in each candidate presentation book image either a set of morphemes included in each candidate presentation book image or a set of morphemes consisting solely of words that have independent meaning from among the morphemes included in said set, and uses a set of morphemes commonly included in the morpheme sets of the plurality of candidate presentation book images as the representative text.

[0128] Step 113: The presentation image acquisition unit 3 obtains, for combinations of Q candidate presentation images ( Ｒ C Ｑ Q combinations out of R candidate presentation images), the total value or average value of similarities between representative texts of all pairs included in each Q combination. As the similarity between representative texts of a pair, a known similarity may be used. For example, the Jaccard coefficient may be used to represent the degree of overlap between word sets appearing in two representative texts, or for each representative text, a text-derived vector may be obtained using generative AI provided in another device, and the cosine similarity or Euclidean distance between the vectors derived from the two representative texts may be used. Of course, as the similarity between representative texts of a pair, vector similarity based on TF-IDF values (for example, cosine similarity or Euclidean distance) may also be used.

[0129] Step 114: The presentation image acquisition unit 3 sets, among Ｒ C Ｑ Q combinations, the Q combination having the smallest total value or average value of the similarities obtained in step 113 as the Q presentation images.

[0130] In the fourth method, the presentation image acquisition unit 3 may use object set similarity instead of text similarity. In this case, the presentation image acquisition unit 3 should replace "text" in steps 112 and 113 with "object set" and perform steps 111 to 114. Technically, the object recognition that obtains the object set corresponds to the process of obtaining morphemes in step 112.

[0131] In the fourth method, the presentation image acquisition unit 3 may, in step 113, obtain both text similarity and object set similarity, and instead of using representative text similarity, use similarity where the text similarity is greater and object set similarity is greater, and perform steps 113 and 114. In short, in the fourth method, the presentation image acquisition unit 3 obtains Q presentation images by selecting Q images with large differences in motifs from R candidate presentation images.

[0132] In the fourth method, the Q presentation images obtained in step 114 of the presentation image acquisition unit 3 become the S presentation images output to the image selection control unit 4.

[0133] <Display Unit 5, Input Reception Unit 6> The display unit 5 is a general-purpose display device that displays images output by the image selection control unit 4 and the search result image generation unit 9.

[0134] The display unit 5, for example, displays a presentation image in which presentation book images with the same motif but different artistic styles are arranged in different positions.

[0135] The input receiving unit 6 is a general-purpose input device that receives input operations for any position in the image displayed by the display unit 5 and outputs information of the input operation to the image selection control unit 4 or the presentation image acquisition unit 3. It is an input device that operates in cooperation with a general-purpose display device.

[0136] The input receiving unit 6 accepts a selection input operation to select a book image from among multiple book images included in the presentation image that has a style preferred by the target person.

[0137] Examples of the display unit 5 and input receiving unit 6 include a touchscreen, a display device without the input receiving unit 6 (a so-called "display") and a mouse, etc.

[0138] The image acquisition device 10 does not necessarily have to include a display unit 5 and an input receiving unit 6. In other words, the display unit 5 and the input receiving unit 6 may be implemented by devices other than the image acquisition device 10. Examples of devices other than the image acquisition device 10 include information terminals such as PCs, smartphones, and tablets owned by the user.

[0139] <Image Selection Control Unit 4> The Image Selection Control Unit 4 receives S (where S is an integer of 1 or more) presentation images as input. The S presentation images input to the Image Selection Control Unit 4 may be obtained by the Presentation Image Acquisition Unit 3 or by other means. However, each presentation image is an image in which multiple presentation book images are arranged in different positions, and if the value of S is 2 or more, the motifs of the S presentation images are different from each other. For example, 3 may be used as the value of S.

[0140] Furthermore, for each of the S presentation images, the image selection control unit 4 receives information from the input reception unit 6 indicating which of the multiple presentation book images included in that presentation image has been selected.

[0141] The image selection control unit 4 obtains, for each of the S presentation images, a presentation book image corresponding to the information indicating which of the multiple presentation book images contained in the presentation image was selected, as a search query image (step S4). The image selection control unit 4 outputs the S search query images to the search unit 8.

[0142] Let s be an integer between 1 and S, and let each of the S presentation images be the s-th presentation image. The image selection control unit 4, the display unit 5, the input reception unit 6, and the user perform steps 201 to 208 below to obtain a presentation book image corresponding to the selection information for each of the S presentation images as a search query image. The process of steps 201 to 208 will be explained with reference to Figure 3.

[0143] Step 201: The image selection control unit 4 sets the value of s to 1.

[0144] Step 202: The image selection control unit 4 controls the display unit 5 so that the sth presentation image is displayed.

[0145] Step 203: The display unit 5 displays the s-th presentation image in accordance with step 202. That is, the display unit 5 presents the s-th presentation image to the user.

[0146] Step 204: The user performs a selection input operation to choose the book image with the art style they prefer most from among the multiple book images included in the s-th presentation image displayed in step 203.

[0147] If the display unit 5 and input receiving unit 6 are touchscreens, the user should simply touch the book image in the style they prefer most. If the display unit 5 is a so-called "display" and the input receiving unit 6 is a mouse, the user should display the mouse cursor on the book image in the style they prefer most and then click the mouse.

[0148] The user may be the same person as the subject, for example, a young child, or a caregiver, for example, a parent of the subject. If the user is a caregiver, the user should show the subject the s-th presentation image presented in step 203, and then ask the subject which presentation book image has their preferred art style, and then perform the selection input operation.

[0149] Step 205: The input receiving unit 6 receives the selection input operation from step 204 and outputs selection information corresponding to the selection input operation from step 204 to the image selection control unit 4. The selection information corresponding to the selection input operation from step 204 is, for example, information representing the position on the screen that was the target of the selection input operation from step 204.

[0150] Step 206: The image selection control unit 4 identifies which of the multiple presentation book images contained in the s-th presentation image corresponds to the selection information from step 205, and obtains the identified presentation book image as the s-th search query image.

[0151] The book image corresponding to the selection information among the multiple book images included in the s-th presentation image is the book image with the art style most preferred by the subject among the multiple book images included in the s-th presentation image. Therefore, the s-th search query image obtained in step 206 is the book image with the art style most preferred by the subject among the multiple book images included in the s-th presentation image.

[0152] Step 207: The image selection control unit 4 terminates the series of processes if s is equal to S, and proceeds to step 208 if s is less than S.

[0153] Step 208: The image selection control unit 4 adds 1 to the current value of s and returns to step 202.

[0154] By following steps 201 to 208 above, S search query images are obtained.

[0155] As can be seen from the description of the material acquisition unit 2 for presentation, in each s, the multiple book images for presentation included in the presentation image presented in step 203 must satisfy at least the following condition A1, and may also satisfy the following condition A2.

[0156] Condition A1: Multiple book images for presentation are multiple images with the same motif but different artistic styles. For example, multiple book images for presentation are multiple images with a high degree of similarity in the group of objects that make up the included picture, or multiple images with the same title but different in at least one of the unique information, author information, or version information.

[0157] Condition A2: The multiple book images for presentation are multiple (N) images of the same motif but with different artistic styles. These N images were selected from a larger number of candidate images (K) and are those with a large difference in similarity based on image features. The K candidate images are also K images of the same motif but with different artistic styles.

[0158] Furthermore, as can be seen from the description of the presentation image acquisition unit 3, when S is a value of 2 or more, the multiple presentation images presented sequentially by multiple steps 203 are multiple presentation book images arranged in different positions from each other, satisfying at least the following condition B1, and may also satisfy the following conditions B2 and / or B3.

[0159] Condition B1: Each presentation image contains multiple images with the same motif but different artistic styles, and the multiple presentation images each depict a different motif.

[0160] Condition B2: Presentation book images placed in the same position across multiple presentation images show significant differences in image features.

[0161] Condition B3: Let Q be the number of images to be presented, and R be an integer greater than Q. Each of the R candidate images to be presented contains multiple candidate book images with the same motif but different artistic styles. The Q images to be presented are those selected from the R candidate images that have a large difference in motifs.

[0162] Furthermore, as can be seen from the description of the image acquisition unit 3 for presentation, the one or more images presented in step 203 may satisfy the following condition C1.

[0163] Condition C1: The presentation image is selected from a larger number of candidate presentation images, based on the user's input regarding the subject's attributes.

[0164] In step 204, the user may perform a selection input operation to select multiple book images with a preferred art style from among the multiple book images included in the s-th presentation image displayed in step 203. In this case, in step 206, multiple search query images will be obtained from the s-th presentation image.

[0165] <Searchable book database 7, search unit 8, search result image generation unit 9> The searchable book database 7 stores multiple books, each associated with a page image and information that identifies the book.

[0166] The search unit 8 receives S search query images as input. The search unit 8 performs an image similarity search using the S search query images as queries from the search book DB 7 to obtain information identifying books that contain images similar to the search query images, and outputs this information to the search result image generation unit 9 (step S8). Any well-known technique may be used for the image similarity search; for example, Google® Image Search may be used.

[0167] For example, the technology described in Japanese Patent Publication No. 2017-129901 can be used as an image similarity search technique for searching for books.

[0168] When the technology described in Japanese Patent Publication No. 2017-129901 is used, the searchable book DB 7 further stores multiple image feature quantities corresponding to the representative image of each of the multiple books. The representative image of a book is an image of a predetermined page that contains a picture from among the multiple pages that make up that book.

[0169] In this case, the search unit 8 extracts multiple image features from the search query image and uses the extracted image features corresponding to the search query image and the image features corresponding to the representative images of the multiple books read from the search book DB 7 to search among the multiple books for a set of one or more books that have a representative image with high similarity to the search query image. At that time, along with the set of books, a score representing the rank or similarity of each book that makes up the set of books is obtained. The score representing the rank or similarity of each book that makes up the set of books is abbreviated as ranking information. The search unit 8 performs this search process for each of the S search query images to obtain S pairs (set of books, ranking information) consisting of a set of books and ranking information.

[0170] The search unit 8 then integrates S pairs (sets of books, ranking information) using a predetermined Rank Fusion method to ultimately obtain information that identifies a book. As the Rank Fusion method, for example, CombSUM, Minimax Rank Fusion, or Reciprocal Rank Fusion (RRF) can be used.

[0171] CombSUM is a method that sums the scores in each ranking and sorts them in descending order of total value. Minimax Rank Fusion is a method that takes the highest rank (worst rank) in each ranking and compares these to determine the ranking. Reciprocal Rank Fusion (RRF) is a method that calculates a score of 1 / (k+r) for each rank r and sorts them in descending order of total value. k is a predetermined adjustment constant. k is determined appropriately to obtain the desired result.

[0172] The search result image generation unit 9 generates an image that displays information identifying the book in a desired display format and outputs it to the display unit 5 (step S9).

[0173] [Example of User Interface] The following describes an example of a user interface displayed on the display unit 5. In this example, the display unit 5 is a touchscreen that also functions as an input receiving unit 6.

[0174] First, the display shown in Figure 4 is displayed. The display in Figure 4 includes the text "How old are you?" and icons indicating age. When the user touches one of the age icons, the display shown in Figure 5 is displayed.

[0175] The display in Figure 5 includes the text "Which one are you?", an icon indicating gender, and an icon indicating skip. When the user touches any of the icons, the display in Figure 6 is displayed.

[0176] Figure 6 displays the text "Tell me your favorite color" and icons representing colors. The multiple hatches shown in Figure 6 each represent a different color. When the user touches any of the icons, Figure 7 is displayed.

[0177] Figure 7 displays the text "What kind of picture book would you like to read?" and icons indicating book genres. When the user touches any of the icons, Figure 8 is displayed.

[0178] At least one of the pieces of information selected by touch in the display from Figures 4 to 7 may be output to the presentation image acquisition unit 3 and used as an attribute in the processing of the presentation image acquisition unit 3.

[0179] The display in Figure 8 includes a presentation image containing six book images featuring a "bear" motif. When the user touches any of the book images, the display in Figure 9 is shown.

[0180] The display in Figure 9 includes a presentation image containing six book images with a "face" motif. When the user touches any of the book images, the display in Figure 10 is shown.

[0181] Figure 10 displays a presentation image containing six book images based on the "Three Little Pigs" story. Users can touch any of the book images.

[0182] In the displays shown in Figures 8 to 10, the book image selected by the user's touch is designated as the search query image.

[0183] [Variations] The specific configuration of the embodiments of the disclosed technology is not limited to the configuration described above. The specific configuration of the embodiments of the disclosed technology can be modified as appropriate, without departing from the spirit of the embodiments of the disclosed technology.

[0184] The various processes described in the embodiments of the disclosed technology may be performed not only in chronological order according to the order described, but also in parallel or individually as required by the processing capacity of the device performing the processes.

[0185] For example, the image acquisition device 10 may be composed of multiple devices capable of sending and receiving data from each other. For instance, the book database 1, the presentation material acquisition unit 2, the presentation image acquisition unit 3, the image selection control unit 4, the display unit 5, and the input reception unit 6 may be provided in any of the multiple devices.

[0186] Data exchange between components of the image acquisition device 10 may be performed directly, or it may be performed via a storage unit (not shown).

[0187] Furthermore, the present invention may also include a device (terminal) for using the apparatus, system, or method of the present invention via a network (telecommunication line). The "device (terminal) for use" may be equipped with functions necessary to obtain the effects of implementing the apparatus, system, or method of the present invention (for example, control functions, decoding functions, restoration functions, input / output functions, etc.).

[0188] It goes without saying that the invention may be modified as appropriate without departing from its spirit.

[0189] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually described as being incorporated by reference.

[0190] [Programs, Recording Media] The functions realized by the components described herein may be implemented in a circuitry or processing circuitry, including a general-purpose processor, an application-specific processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), a CPU (a Central Processing Unit), conventional circuits, and / or a combination thereof, programmed to realize the functions described herein. A processor includes transistors and other circuits and is considered a circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in memory.

[0191] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0192] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0193] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 11, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0194] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0195] A program describing this process may be included in a computer program product.

[0196] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0197] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed using a so-called ASP (Application Service Provider) type service, where the processing function is realized only by issuing execution instructions and obtaining results, without transferring the program from the server computer to this computer.In addition, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. Furthermore, the term "program" in this form includes information used for processing by an electronic computer that is equivalent to a program (data, etc., that is not a direct instruction to the computer but has the property of defining the processing of the computer).

[0198] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware.

[0199] [Note] The image acquisition device 10 described above can be expressed as follows.

[0200] (1)(01-1) A device for obtaining N book images to present, in order to allow a user to select one or more images from N book images to present, comprising: a book database in which images contained in one or more pages contained in each of a plurality of books are stored; and a presentation material acquisition unit that obtains images of one page contained in each of the N books from the book database to make N book images to present, wherein the presentation material acquisition unit obtains N presentation images by processing which pages contained in each book stored in the book database have high similarity of the text contained in the page and / or similarity of the group of objects constituting the picture contained in the page.

[0201] (2)(01-1-1A) The apparatus of (01-1), wherein the book DB stores the text and images contained in one or more pages contained in each of the multiple books, and the presentation material acquisition unit obtains N presentation book images by selecting the text contained in one page of a first book stored in the book DB as the text to be evaluated for similarity, and selecting a predetermined number of images of pages with the highest degree of similarity between the text contained in the page and the text to be evaluated for similarity from among the pages contained in each of the books other than the first book stored in the book DB, in which case the images of the most similar pages are obtained.

[0202] (3)(01-1-1B) The apparatus of (01-1), wherein the book DB stores the text and images contained in each of the multiple books, for each of the multiple books, the presentation material acquisition unit performs the following processes: The text contained in one page of the first book stored in the book DB is set as the first similarity evaluation target text, M is a value of N or more, and m is an integer from 2 to M-1 in order, and the image of the page in each of the books stored in the book DB that are not the first book to the m-1 book that has the greatest similarity between the text contained in the page and the first similarity evaluation target text and the m-th similarity evaluation target text; The process of selecting the image of the page in each of the books stored in the book DB that are not the first book to the M-1 book that has the greatest similarity between the text contained in the page and the first similarity evaluation target text and the M-1 similarity evaluation target text; The process includes obtaining N images of the aforementioned books for presentation.

[0203] (4)(01-1-2A) The apparatus of (01-1), wherein the presentation material acquisition unit obtains a set of objects contained in an image of a page contained in a first book stored in the book DB by performing object recognition on the image, and obtains a set of objects to be evaluated for similarity by performing object recognition on each candidate image of an image of a page contained in each book other than the first book stored in the book DB, and obtains a set of objects contained in each candidate image as a set of candidate image objects, and obtains N presentation book images by performing a process that includes selecting a predetermined number of images of pages from among the pages contained in each book other than the first book stored in the book DB, from the set with the greatest similarity between the set of candidate image object sets of pages and the set of objects to be evaluated for similarity, in which case the size of the similarity between the sets is the largest.

[0204] (5)(01-1-2B) The apparatus of (01-1), wherein the presentation material acquisition unit performs object recognition on an image of a certain page contained in the first book stored in the book DB, thereby obtaining a set of objects contained in the image as the first similarity evaluation target object set, takes images of pages contained in each book other than the first book stored in the book DB as candidate images, performs object recognition on each candidate image, thereby obtaining a set of objects contained in each candidate image as each candidate image object set, takes one value of N or more as M, and each integer from 2 to M-1 in order as m, and selects the image of the page from among the page candidate image object sets contained in each book other than the first book to the m-1 similarity evaluation target object set that has the largest sum or average value of inter-set similarity between the first similarity evaluation target object set and the m-1 similarity evaluation target object set, and sets the image of that page as the m-th similarity evaluation target object set, The process includes selecting the image of the page from among the pages in any of the books stored in the book database, from the first book to the M-1 book, that has the largest sum or average value of inter-set similarity between the first similarity evaluation target object set and the M-1 similarity evaluation target object set, thereby obtaining N images of the book for presentation.

[0205] (6)(01-2) A device for obtaining N book images to present, in order to have a target person select one or more images from N book images to present, comprising: a title, unique information and / or author information and / or version information, a cover image, a stored book database, and a presentation material acquisition unit that obtains the cover images contained in each of the N books from the book database to make N book images to present, wherein the presentation material acquisition unit obtains N book images to present through a process that includes selecting the cover images of M books (M is an integer of N or more) from the book database that have the same title but differ in at least one of the unique information, author information and version information.

[0206] (7)(01-3) The apparatus of (01-1), wherein the presentation material acquisition unit selects from among the pages contained in each book stored in the book DB that have high similarity of the text contained in the page and / or similarity of the group of objects that constitute the picture contained in the page, thereby obtaining K images of one page contained in each of K books (K is an integer greater than N) as K intermediate candidate book images for presentation, and then obtains N presentation book images by selecting N images from the K intermediate candidate book images for presentation that have large differences in the degree of similarity between images based on image features.

[0207] (8)(01-3A) The apparatus of (01-1), wherein the presentation material acquisition unit selects from among the pages contained in each book stored in the book DB that have high similarity of the text contained in the page and / or similarity of the group of objects that constitute the picture contained in the page, thereby obtaining an image of one page contained in each of the K books (K is an integer greater than N) as K intermediate candidate book images for presentation, and sets of all N intermediate candidate book images contained in the K intermediate candidate book images for presentation ( Ｋ C Ｎ For each of the N pairs of images, the sum or average of the similarity values ​​based on the image features of all pairs included in each N pair is obtained, and the images in the N pair with the smallest sum or average similarity value are selected as the N images to be presented as books.

[0208] (9)(01-3B) The apparatus of (01-1), wherein the presentation material acquisition unit obtains K intermediate candidate book images for presentation by prioritizing the selection of pages from each book stored in the book DB that have high similarity of the text contained in the page and / or similarity of the group of objects constituting the picture contained in the page, thereby obtaining K intermediate candidate book images for presentation from each of the K books (where K is an integer greater than N), obtaining one of the K intermediate candidate book images for presentation as the first presentation book image, and then, by performing a process to select the intermediate candidate book image for presentation that is not any of the first presentation book image to the (n-th)th presentation book image, where n is an integer from 2 to N in order, the intermediate candidate book image for presentation that is not any of the first presentation book image to the (n-th)th presentation book image, and the sum or average value of the similarity in pairs of images with each of the first presentation book image to the (n-th)th presentation book image, thereby obtaining N presentation book images.

[0209] (10)(01-3') The apparatus of (01-2) wherein the presentation material acquisition unit obtains K images of one page contained in each of K books (K is an integer greater than N) from the book DB, by selecting books with the same title but different in at least one of the unique information, author information, and version information, and obtains K intermediate candidate book images for presentation. From the K intermediate candidate book images for presentation, the process of selecting N images with large differences in similarity based on image features obtains N book images for presentation.

[0210] (11)(01-3'A) The apparatus of (01-2), wherein the presentation material acquisition unit selects from the book DB books that have the same title and at least one of the unique information, author information, and version information is different, thereby obtaining one page image from each of the K books (K is an integer greater than N) as K intermediate candidate book images for presentation, and sets of all N intermediate candidate book images for presentation contained in the K intermediate candidate book images for presentation ( Ｋ C ＮFor each of the N pairs of images, the sum or average of the similarity values ​​based on the image features of all pairs included in each N pair is obtained, and the images in the N pair with the smallest sum or average similarity value are selected as the N images to be presented as books.

[0211] (12)(01-3'B) The apparatus of (01-2) obtains N presentation material acquisition units by selecting from the book DB books that have the same title and differ in at least one of the unique information, author information, and version information, thereby obtaining K intermediate candidate book images for presentation, one image of a page contained in each of the K intermediate candidate book images for presentation, and obtaining one of the K intermediate candidate book images for presentation as the first presentation book image, and by performing a process to select the intermediate candidate book image for presentation that is not one of the first presentation book image to the (n-th)th presentation book image, where n is an integer between 2 and N, in order, the intermediate candidate book image for presentation that is not one of the first presentation book image to the (n-th)th presentation book image, and the sum or average value of the similarity in pairs of the first presentation book image to the (n-th)th presentation book image and each of the first presentation book image to the (n-th)th presentation book image is smallest, thereby obtaining N presentation book images.

[0212] (13)(01-3'') A device for obtaining N book images to present, in order to have a subject select one or more images less than N from N book images to present, the device includes a presentation material acquisition unit that obtains N book images to present by selecting N images with large differences in similarity based on image features from K candidate book images to present, which are images of pages contained in each of K books (K is an integer greater than N).

[0213] (14)(01-3''A) A device for obtaining N presentation book images to present in order to have a subject select one or more images less than N from N presentation book images, wherein the device obtains a set of all N presentation candidate book images included in K presentation candidate book images, each of which is an image of a page contained in K books (K is an integer greater than N) ( Ｋ C ＮThe system includes a presentation material acquisition unit that, for each N-group of images, obtains the sum or average of the similarity values ​​based on the image features of all pairs contained in each N-group, and selects the image contained in the N-group with the smallest sum or average similarity value as one of the N presentation book images.

[0214] (15)(01-3''B) A device for obtaining N book images to present, for the purpose of having a subject select one or more images less than N from N book images to present, the device includes a presentation material acquisition unit that obtains one of K candidate book images to present, which are images of pages contained in each of K books (K is an integer greater than N), as the first book image to present, and selects the candidate book image to present, which is one of the candidate book images to present that is not any of the first to (n-th)th book images, where n is an integer between 2 and N, and the sum or average value of the similarity in pairs of the first to (n-th)th book images and each of the first to (n-th)th book images, as the nth book image to present.

[0215] (16)(01-4) A device for obtaining S presentation images (S is an integer of 2 or more) in order to have a user select one or more but less than N images from N presentation book images (N is an integer of 2 or more) that have the same motif but different drawing styles, wherein the S presentation images (S is an integer of 2 or more) are arranged in predetermined N positions, and the set of N presentation book images (N have the same motif but different drawing styles) is called a presentation book image set, and each presentation image has predetermined N positions common to the S presentation images, and the device includes a presentation image acquisition unit that receives S presentation book image sets (S have different motifs) as input and arranges the N presentation book images included in each presentation book image set in predetermined N positions within each presentation image to obtain S presentation images, wherein n is an integer of 1 or more and less than N, and each predetermined N position is a predetermined nth position, and the presentation image acquisition unit arranges the images so that for all n, the degree of similarity based on the image features of the S presentation book images arranged in the predetermined nth position in the S presentation images is small. Place the N images of the book being presented, which are included in each set of book images, within each image.

[0216] (16A)(01-4-1) An image to be presented to a user in order to have the user select one or more but less than N images from N images of a book to be presented that have the same motif but different styles, the device to obtain S images of a book to be presented that have the same motif but different styles, with the N images arranged in predetermined N positions, wherein a set of N images of a book to be presented that have the same motif but different styles is called a set of book to be presented, and each image has predetermined N positions common to the S images, and the device includes a set of S sets of book to be presented that have different motifs as input, and obtains S images of a book to be presented by randomly arranging the N images included in each set of book to be presented in predetermined N positions within each image.

[0217] (17)(01-5) An image to be presented to a user in order to have the user select one or more but less than N images from N presentation book images (N is an integer of 2 or more) that have the same motif but different drawing styles, and a device for obtaining a presentation image that includes N presentation book images, which, based on the attributes of the input subject, obtains one of the input multiple presentation candidate images as the presentation image, each presentation candidate image includes N presentation candidate book images that have the same motif but different drawing styles, the motifs of the multiple presentation candidate images are different from each other, each presentation candidate image is associated with information that identifies the motif and / or information that represents the degree of suitability to the attributes, and the presentation image acquisition unit prioritizes the presentation candidate image with a high degree of suitability to the attributes of the subject from among the multiple presentation candidate images and uses it as the presentation image.

[0218] (17A)(01-6) An image present to a user in order to have the user select one or more images less than N from N images of a book to be presented that have the same motif but different drawing styles, and the device obtains Q images of presentation that contain N images of a book to be presented, the device includes an image acquisition unit that obtains Q images of presentation candidates from R images of presentation candidates that are input (R is an integer greater than Q), each of which includes N images of a book to be presented that have the same motif but different drawing styles, and the image acquisition unit obtains Q images of presentation by selecting Q images of which have a large difference in motif from the R images of presentation candidates.

[0219] (18)(02-1) A method for obtaining a search query image of a preferred style of drawing by a target person, comprising: a display step in which a display unit presents a presentation image in which presentation book images of the same motif but with different styles are arranged in different positions; an input receiving step in which an input receiving unit receives a selection input operation to select a presentation book image of a preferred style of drawing by a target person from among a plurality of presentation book images included in the presentation image; and an image selection control step in which an image selection control unit obtains a presentation book image corresponding to the selection input operation received by the input receiving unit from among a plurality of presentation book images included in the presentation image as a search query image.

[0220] (19)(02-1A) The method of (02-1), wherein the multiple book images for presentation are multiple images with high similarity in the group of objects that make up the included picture, and / or multiple book images for presentation that have the same title but differ in at least one of the unique information, author information, and version information.

[0221] (20)(02-1-1) The method of (02-1) is used to obtain multiple search query images by sequentially using multiple presentation images to perform a display step, an input reception step, and an image selection control step.

[0222] (21)(02-1-1-1) The method of (02-1-1), wherein the motifs in the multiple presentation images are different from each other.

[0223] (22)(02-1-1-2) In the method of (02-1-1), the presentation book images that are placed in the same position in multiple presentation images have large differences in their image features.

[0224] (23)(02-1-2) The method of (02-1), wherein the multiple candidate images are images with the same motif but different artistic styles, and the multiple presentation book images included in the presentation image are images selected from a larger number of candidate images that have a large difference in similarity based on image features.

[0225] (24)(02-1-3) The method of (02-1-1), further comprising an attribute input reception step in which an input reception unit receives a selection input operation in which an attribute of the subject is selected, wherein the presentation image presented by the display unit in the display step is selected from among the candidate presentation images in accordance with the selection input operation.

[0226] (25)(02-1-4) The method of (02-1-1), wherein Q is an integer of 2 or more which is the number of images to be presented, and R is an integer greater than Q, and each of the R candidate images to be presented contains multiple candidate book images that have the same motif but different styles, and the Q images to be presented are the Q images selected from the R candidate images to be presented that have a large difference in motif.

Claims

1. An image acquisition method comprising: a display step in which a display unit presents a presentation image in which presentation book images with the same motif but different drawing styles are arranged in different positions; an input receiving step in which an input receiving unit receives a selection input operation to select a presentation book image with a drawing style preferred by the target person from among a plurality of presentation book images included in the presentation image; and an image selection control step in which an image selection control unit obtains a presentation book image corresponding to the selection input operation received by the input receiving unit from among a plurality of presentation book images included in the presentation image as a search query image.

2. The image acquisition method of claim 1, wherein the plurality of presentation book images are a plurality of images with high similarity in the group of objects constituting the included picture, and / or a plurality of presentation book images with the same title but different in at least one of the unique information, author information, and version information.

3. An image acquisition method according to claim 1, wherein multiple search query images are obtained by sequentially using multiple presentation images to perform a display step, an input reception step, and an image selection control step, and the motifs in the multiple presentation images are different from each other.

4. An image acquisition method according to claim 1, wherein multiple search query images are obtained by sequentially using multiple presentation images to perform a display step, an input reception step, and an image selection control step, and presentation book images placed at the same position among the multiple presentation images have large differences in their image features.

5. An image acquisition method according to claim 1, further comprising an attribute input reception step in which an input reception unit receives a selection input operation for selecting the attributes of a subject, wherein the image to be presented by the display unit in the display step is selected from among candidate images to be presented according to the selection input operation.

6. An image acquisition method according to claim 1, wherein multiple search query images are obtained by performing a display step, an input reception step, and an image selection control step using multiple presentation images in sequence, where Q is the number of presentation images, and R is an integer greater than Q, where each of the R presentation candidate images contains multiple presentation candidate book images with the same motif but different drawing styles, and the Q presentation images are the Q images selected from the R presentation candidate images that have a large difference in motif.

7. An image acquisition device comprising: a display unit that displays a presentation image in which presentation book images with the same motif but different artistic styles are arranged in different positions; an input receiving unit that receives a selection input operation to select a presentation book image with an artistic style preferred by the user from among multiple presentation book images included in the presentation image; and an image selection control unit that obtains a presentation book image corresponding to the selection input operation received by the input receiving unit from among multiple presentation book images included in the presentation image as a search query image.

8. A program for causing a computer to perform each step of the image acquisition method according to any one of claims 1 to 6.

Images