Information processing device, control method for same, and program

Abstract

This information processing device comprises: a conversion unit that converts a first image into a second image by maintaining the aspect ratio of the first image and changing the resolution thereof such that the vertical or horizontal dimension of the first image matches the vertical or horizontal dimension of a display region; a position control unit that controls the position of the second image obtained by conversion from the first image by the conversion unit in the display region; a generation unit that generates an expanded image of the second image by generating a third image for a margin region of the display region that does not overlap the second image the position of which is controlled by the position control unit; and an output unit that outputs data of the expanded image generated by the generation unit.

Description

Information Processing Apparatus, Control Method Thereof, and Program

[0006] ,

[0005] ,

[0001] The present disclosure relates to an information processing apparatus, a control method thereof, and a program.

[0002] When displaying an image captured by a camera on a display of another device, or when rotating the portrait or landscape orientation of a smartphone when displaying an image recorded on the smartphone, the aspect ratio of the display target image and the display area may not match. In such a case, if the resolution of the display target image is changed so as to match the vertical or horizontal length of the display area while maintaining the aspect ratio of the display target image, a blank area where the display target image does not exist will occur in the display area. In such a case, a technique of displaying a black band in the blank area is known. Also, techniques such as blurring and copying the edges of the display target image and displaying the copied image in the blank area, or displaying other content in the blank area are generally known. Further, in Patent Document 1, a technique of trimming the display target image according to the aspect ratio of the display area and displaying the trimmed image over the entire display area is described in order not to generate a blank area.

[0003] Japanese Patent Application Laid-Open No. 9-****

[0004] However, in the above conventional techniques, it is considered that the image displayed in the blank area is not recognized as having continuity or consistency with the display target image. Therefore, it may cause a sense of discomfort to the user, and as a result, the user's satisfaction may decrease. Also, in the technique described in Patent Document 1, the entire display target image is not displayed. Such a situation can also be a factor in reducing the user's satisfaction.

[0005] The present disclosure has been made in view of at least one of the above problems, and provides a display technique for a display target image that suppresses a decrease in user satisfaction when the aspect ratio of the display target image and the display area is different.

[0006] One aspect of this disclosure is an information processing device comprising: a conversion unit that converts a first image into a second image by maintaining the aspect ratio of the first image and changing the resolution so that the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of a display area; a position control unit that controls the position of the second image converted from the first image by the conversion unit in the display area; a generation unit that generates an extended image of the second image by generating a third image in the blank area of ​​the display area that does not overlap with the second image whose position is controlled by the position control unit; and an output unit that outputs the data of the extended image generated by the generation unit.

[0007] According to this disclosure, it is possible to provide a display technology for a target image that suppresses a decrease in user satisfaction when the aspect ratio of the target image and the display area are different.

[0008] Other features and advantages of the technical ideas derived from this disclosure will become apparent from the following description with reference to the attached drawings. In the attached drawings, the same or similar components are given the same reference numeral.

[0009] The attached drawings are included in the specification and constitute part thereof, illustrating embodiments in this disclosure and are used together with the description to explain the technical ideas derived from this disclosure.

[0010] Block diagram of a display device according to an embodiment Block diagram of a display device according to an embodiment Function of the image generation unit according to an embodiment Block diagram of a flowchart of processing according to an embodiment Figure 1 showing a display screen Function of the image generation unit according to an embodiment Block diagram of processing according to an embodiment Block diagram of a display device according to an embodiment Figure 1 showing a display screen according to an embodiment Figure 1 showing a display screen according to an embodiment Figure 1 showing a display screen according to an embodiment Figure 1 showing a display screen according to an embodiment Figure 1 showing a display screen according to an embodiment Figure 1 showing a display screen according to an embodiment Figure 1 showing a display screen according to an embodiment Figure 1 showing a flowchart of processing according to an embodiment Figure 1 shows a display screen relating to the embodiment.

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claims. While the embodiments describe multiple features, not all of these features are necessary, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] (First Embodiment) <Example of Hardware Configuration of Display Device> The hardware configuration of the display device 100 (an example of an "information processing device") will be explained using Figure 1B. The display device 100 is composed of a CPU (Central Processing Unit) 112, ROM 113, RAM 114, and recording medium 115. The display device 100 is also composed of a GPU (Graphics Processing Unit) 116, input device 117, display device 118, and communication interface 119. Each component is connected to each other so as to be able to communicate with each other via a system bus 111.

[0013] The CPU 112 and GPU 116 are each one or more processors capable of executing programs. The GPU 116 is configured to perform certain operations faster than the CPU 112, and in recent years has been used particularly to perform inference processing using neural networks at high speed. As an alternative to the GPU 116, an NPU (Neural Processing Unit) that is more specialized for performing learning and inference processing using neural networks may be provided. Alternatively, instead of using separate CPU 112 and GPU 116, a System on Chip (SoC) that integrates the CPU and GPU (and possibly an NPU) may be provided.

[0014] Furthermore, the CPU 112 implements each functional block described later in Figures 1A and 2 by, for example, loading a program stored in ROM 113 into RAM 114 and executing it. The CPU 112 achieves high-speed processing by utilizing GPU 116 for processing using neural networks. Note that GPU 116 may have its own dedicated RAM separate from RAM 114.

[0015] ROM 113 is, for example, an electrically rewritable non-volatile memory. ROM 113 stores programs that the CPU 112 can execute, setting values, GUI data, and parameters for implementing a trained neural network (training model). RAM 114 is used to load programs executed by the CPU 112 and to store values ​​necessary during program execution. Furthermore, RAM 114 includes VRAM (Video RAM) in part for display control. The recording medium 115 is, for example, a semiconductor memory card or an SSD (Solid State Drive), and is also used as a recording destination for input image data of the display device 100.

[0016] The input device 117 includes multiple operating parts, such as buttons, touch panels, or switches, that receive operational input to the display device 100. The input device 117 may also include one or more devices (such as sensors) for acquiring additional information during image generation. That is, the input device 117 may include, for example, a GPS receiver for acquiring the location information of the display device 100, or a clock for acquiring the generation date and time. However, the input device 117 is not limited to these devices.

[0017] The display device 118 is, for example, a liquid crystal display provided on the surface of the housing of the display device 100. The display device 118 may also be a touch display. The display device 118 displays, for example, image data drawn in the VRAM within the RAM 114, a menu screen, and information about the display device 100 (e.g., setting values, battery level, etc.).

[0018] The communication interface 119 is a circuit for communicating with an external device in accordance with one or more communication standards, and includes a connector for wired communication, an antenna for wireless communication, and a transmitting and receiving circuit. The display device 100 can transmit image data to an external device and receive data from an external device through this communication interface 119. Typical communication standards to which the communication interface 119 conforms include, but are not limited to, HDMI®, USB, Bluetooth (Registered Publication), and Wireless LAN (Wi-Fi).

[0019] <Example of Functional Configuration of Display Device> Next, the functional blocks of the display device 100 will be described using Figure 1A. Throughout the drawing, functional blocks can be implemented by software, or a combination of software and hardware, except for parts that are clearly only achievable by hardware (e.g., recording media). For example, a functional block may be implemented by dedicated hardware such as an ASIC. Alternatively, a functional block may be implemented by one or more program-executable processors, such as a CPU, executing a program stored in memory. Note that multiple functional blocks may be implemented by a common configuration (e.g., one ASIC). Furthermore, hardware that implements some functions of one functional block may be included in hardware that implements other functional blocks.

[0020] As shown in Figure 1A, the display device 100 is configured to include an image input unit 101, an image generation unit 102, and a display unit 103. The functions of the display device 100 are not limited to those realized by the functional blocks shown in Figure 1A. The image input unit 101 acquires image data recorded on the recording medium of the display device 100 as input image data (an example of a "first image"). Here, the image data may be, for example, an image captured by a digital camera or smartphone.

[0021] Furthermore, the image input unit 101 outputs the resolution of the input image data to the image generation unit 102. In addition, the image input unit 101 may output to the image generation unit 102 some of the supplementary information recorded in the data file in which the input image data is stored (for example, one or more tags related to the shooting conditions and shooting status of the captured image).

[0022] In this embodiment, the example described is when the image input unit 101 acquires image data from a recording medium of the display device 100, but the invention is not limited to this example. For example, the image input unit 101 may acquire image data recorded on an external server or cloud that the display device 100 can communicate with. Alternatively, the image input unit 101 may acquire image data recorded on an external storage device such as a USB memory or external HDD connected to the display device 100.

[0023] The image generation unit 102 generates display image data using input image data acquired from the image input unit 101. The image generation unit 102 then outputs the generated display image data to VRAM. The display unit 103 reads the display image data generated by the image generation unit 102 from VRAM and displays it on the display device.

[0024] Using Figure 2, a block illustrating an example of the functional configuration of the image generation unit 102 will be explained. The image generation unit 102 is composed of a resolution conversion unit 201, an image expansion unit 202, and an image output unit 203. The resolution conversion unit 201 converts the horizontal or vertical dimensions of the input image to the horizontal or vertical dimensions of the display area while maintaining the aspect ratio of the resolution of the input image. The resolution conversion unit 201 then outputs the converted input image data to the image expansion unit 202.

[0025] The image extension unit 202 extends the original image by generating a new image that is continuous or consistent with the image data output from the resolution conversion unit 201, for example, using an image generation AI model. More specifically, the image generation AI model is a neural network generated using, for example, an image cropped from an existing image as input image data, and image data surrounding the cropped image from the existing image as training data. The parameters of this image generation AI model are, for example, image patterns or features. The image generation AI model thus trained can infer patterns or features of surrounding images from the input image and generate a new image around the input image. For the training method and inference process, well-known techniques related to image generation AI may be applied. The image generation AI model may be stored in advance in the display device 100, for example, or stored in an external device that the display device 100 can communicate with. Furthermore, the method for generating the image around the input image is not limited to the above method, and may be a method that generates the image according to a predetermined rule, for example. The image output unit 203 outputs the image data output from the resolution conversion unit 201 or the image extension unit 202 to the VRAM. The input image extended by the newly generated image in this way will hereafter be referred to as the extended image.

[0026] The functional blocks shown in Figures 1A and 2 are implemented by one or more of the hardware components shown in Figure 1B. For example, the image generation unit 102 is mainly implemented by the CPU 112 and GPU 116. The RAM 114 is used as a temporary storage location for data to be processed, data being processed, and processing result data, while the ROM 113 is used as a reference point for pre-stored settings and programs, and is used for various processes.

[0027] <Operation during display processing> The display operation of the display device 100 will be explained using Figures 3 and 4A to 4D. Note that the operation of each step in the flowchart of Figure 3 is realized by the CPU 112 or GPU 116 executing a program stored in ROM 113 and controlling other hardware of the display device 100 as necessary.

[0028] Furthermore, the display operation of the display device 100 is performed in response to instructions received through the input device 117. For example, a list of thumbnails of input images may be displayed on the display device 118, and the system may respond to selection operations via the input device 117. Alternatively, the display operation of the display device 100 may be performed according to predetermined conditions other than instructions. For example, the display operation may be performed sequentially and automatically for multiple image data received from an external device via the communication interface 119.

[0029] In S301, the image input unit 101 acquires input image data. The image input unit 101 then outputs the acquired input image data and the associated information described above to the image generation unit 102. The image input unit 101 may also process the input image data to make it suitable for use by the image generation unit 102 before outputting it. In S302, the resolution conversion unit 201 of the image generation unit 102 acquires the resolution of the input image and the display area, respectively. The resolution conversion unit 201 then converts the resolution of the input image data so that the horizontal or vertical width of the input image matches the horizontal or vertical width of the display area, while maintaining the aspect ratio of the input image. The image after the resolution conversion in this way is an example of the "second image".

[0030] Figure 4A illustrates an input image acquired by the image input unit 101. The aspect ratio of the input image is 16:9. Figure 4B illustrates the display area of ​​the display device 118. The aspect ratio of the display area is 3:2. Figure 4C illustrates an image in which the input image from Figure 4A is placed on the display area of ​​Figure 4B. At this time, the resolution conversion unit 201 performs a conversion that matches the horizontal width of the input image to the horizontal width of the display area while maintaining the aspect ratio of the input image. In this way, when the aspect ratios of the input image and the display area are different, the resolution conversion unit 201 executes S302, which creates areas around the input image where no image exists. In Figure 4C, such areas are exemplified by regions (401, 402) above and below the display area. In the first embodiment, the subsequent processing S303 to S304 generates a new image for regions (401, 402). The input image is expanded in this way.

[0031] Returning to the explanation of the flowchart in Figure 3, in S303, the resolution conversion unit 201 determines whether the aspect ratio of the input image acquired by the image input unit 101 and the display area are different. If the resolution conversion unit 201 determines that the aspect ratios are different, the process proceeds to S304. Otherwise, the resolution conversion unit 201 uses the input image as the display image, and the process proceeds to S305.

[0032] In S304, the image extension unit 202 (an example of a "position control unit") places the input image in the center of the display area, for example, as shown in Figure 4C. This creates areas (401, 402) where the input image does not overlap. The image extension unit 202 then generates a new image (an example of a "third image") that is continuous or consistent with the input image for these areas (401, 402). The image extension unit 202 then combines the input image data and the newly generated image data to generate data for a display image (extended image). This makes it possible to display the image across the entire display area even when the aspect ratio of the input image and the resolution of the display area are different. Figure 4D shows an example of a display image displayed across the entire display area. Note that areas (401, 402) are an example of a "margin area".

[0033] In S305, the image output unit 203 stores the display image data output from the resolution conversion unit 201 in S302 or the image expansion unit 202 in S304 in the VRAM of RAM 114. In S306, the display unit 103 reads the display image data stored in the VRAM of RAM 114 and displays it on the display device 118. Then the process ends.

[0034] <One aspect of its function and effect> As described above, the display device 100 generates a new image that is continuous or consistent with the input image when the aspect ratio of the input image and the display area are different. Then, the input image and the newly generated image are combined to create a display image. In this way, it becomes possible to display the image across the entire display area regardless of the difference in aspect ratio between the input image and the display area. Therefore, a decrease in user satisfaction is suppressed.

[0035] (Second Embodiment) Next, the display device 100 according to the second embodiment will be described. The display device 100 according to the second embodiment allows the user to arbitrarily select a display method when the aspect ratio of the input image and the display area are different. In addition, the operation of the display device 100 when the aspect ratio of the display area is changed during image display will also be described in the second embodiment. Note that in the second embodiment, the processing of the image generation unit 102 differs from that of the first embodiment, but other processing is the same as in the first embodiment. Therefore, the same reference numerals are used for components and processes that are the same as or substantially the same as those described above in the first embodiment, and their descriptions are omitted.

[0036] <Operation during display processing> The display operation of the display device 100 according to the second embodiment will be explained using Figures 5A, 5B, and 6A to 6H. First, an example of a display image will be explained using Figures 6A to 6D. Figure 6A illustrates an input image acquired by the image input unit 101. The aspect ratio of the input image is 16:9. Figure 6B illustrates the display area of ​​the display device 118. The aspect ratio of the display area is 3:2. Figure 6C illustrates a display image in which the horizontal width of the input image in Figure 6A is matched to the horizontal width of the display area in Figure 6B. In the display image of Figure 6C, areas where no image exists occur in the upper and lower areas (601, 602) of the display area, and these areas (601, 602) are filled with black.

[0037] Furthermore, Figure 6D illustrates a display image in which the vertical width of the input image exemplified in Figure 6A is matched to the vertical width of the display area in Figure 6B. In other words, the display image in Figure 6D is a display image in which the aspect ratio of the input image is matched to the aspect ratio of the display area by cropping the left and right regions of the input image. Thus, when the aspect ratio of the input image and the display area are different, methods for displaying the input image include displaying the areas where the input image does not exist as black, or displaying the input image by cropping a part of it and matching the aspect ratio.

[0038] On the other hand, Figures 6E to 6H illustrate the display images generated by each embodiment. Specifically, Figure 6E is a display image (Figure 4D) generated by the image generation process according to the first embodiment. In the second embodiment, in addition to the display methods for the display images (Figures 6C and 6D), the user of the display device 100 can arbitrarily select a display method such as the display image (Figure 6E) generated in the first embodiment.

[0039] Next, a flowchart of the display operation of the display device 100 will be explained using Figures 5A and 5B. Note that the operation of each step in the flowcharts in Figures 5A and 5B is achieved by the CPU 112 or GPU 116 executing a program stored in the ROM 113 and controlling other hardware of the display device 100 as needed.

[0040] Furthermore, the display operation of the display device 100 is performed in response to instructions received through the input device 117. For example, a list of thumbnails of input image data may be displayed on the display device 118, and the system may respond to selection operations via the input device 117. Alternatively, the display operation of the display device 100 may be performed according to predetermined conditions other than instructions. For example, the display operation may be performed sequentially and automatically for multiple image data received from an external device via the communication interface 119.

[0041] Since steps S301 and S302 are the same as in the first embodiment, their explanation is omitted. The process then proceeds to S501. In S501, the resolution conversion unit 201 determines whether the aspect ratio of the resolution of the input image acquired by the image input unit 101 and the aspect ratio of the resolution of the display area are different. If the resolution conversion unit 201 determines that the aspect ratios are different, the process proceeds to S502; otherwise, the process proceeds to S305.

[0042] In S502, the resolution conversion unit 201 (an example of the "first determination unit") determines whether it is possible to generate an enlarged image by the image enlargement unit 202 using an arbitrary set value specified by the user operating the display device 100. When the resolution conversion unit 201 determines that the user has permitted the generation of an enlarged image, the process proceeds to S304 and S305 in this order. Otherwise, the process proceeds to S503.

[0043] In S503, the image output unit 203 determines whether to paint the areas (601, 602) around the input image shown in FIG. 6C black or to trim the input image according to the aspect ratio of the display area and display it in full. Note that an arbitrary set value specified by the user operating the display device 100 is used for this determination. When the image output unit 203 determines to paint it black, the process proceeds to S504. When it determines to trim and display it in full, the process proceeds to S505.

[0044] Note that the method by which the user specifies the set value used for the determinations in S502 and S503 is, for example, when it is determined in S501 that the aspect ratios of the input image and the display area are different after the start of display processing, the following processing is executed. That is, the display unit 103 displays an arbitrary notification on the screen. Then, the CPU 112 accepts that the user makes a specified operation on the content of the notification via an arbitrary operation member.

[0045] More specifically, the display unit 103 displays, for example, a message indicating options for the display method of the display image, such as "Enlarge and display the image", "Paint the periphery of the image black and display it", and "Trim the image and display it", on the touch panel display. The user specifies the display method of the image by selecting one of them via the touch panel. Then, the CPU 112 accepts the selection operation.

[0046] Alternatively, by adding a setting related to the display method to the setting items of the display device 100, a method that can be arbitrarily specified in advance may also be used. Note that the specification of the image display method by the user is not limited to these methods, and other methods may be used for specification. Also, in the second embodiment, three display methods can be arbitrarily selected, but other display methods may also be arbitrarily selectable. Other display methods include, for example, displaying other information in the areas (601, 602) around the input image, or copying the input image edges instead of painting them black and filling the areas (601, 602) with the copied images.

[0047] In S504, the image output unit 203 assigns a black image to the areas (601, 602) where the input image does not exist. FIG. 6C illustrates a display image with a black image assigned to the areas (601, 602). Then, the process proceeds to S305.

[0048] In S505, the image output unit 203 cuts out the input image in accordance with the aspect ratio of the resolution of the display area. Regarding the cut-out position of the input image, the center position of the input image and the display area is made to coincide, and the position is determined such that either the horizontal or vertical width of the input image becomes the same as the horizontal or vertical width of the display area. Alternatively, the cut-out position may be arbitrarily determined by the user. In this way, a display method as shown in FIG. 6D, in which a part of the input image is cut out and adjusted to the aspect ratio of the display area, is realized. Then, the process proceeds to S305.

[0049] Steps S304 to S306 are the same as in the first embodiment, so their explanation is omitted. After the execution of S306, the process proceeds to S506. In S506, the display unit 103 determines whether the display state of the display image has been canceled, that is, whether the display process has ended. If the display unit 103 determines that the display process has ended, the process ends; otherwise, the process proceeds to S507. The conditions under which the display state of the display image is canceled include, for example, when the display device 118 is turned off, or when the display state of the display device 118 transitions to another mode. Another mode may be, for example, a mode in which a list of thumbnail images is displayed, or a mode in which a settings screen for configuring the display device 100 is displayed.

[0050] In S507, the display unit 103 (an example of the "second determination unit") determines whether the aspect ratio of the display area has been changed while the display image is being displayed. If the display unit 103 determines that the aspect ratio has been changed, the process returns to S302; otherwise, the process returns to S506.

[0051] Here, we will explain the details of S507. Figure 6F illustrates the display area of ​​the display device 118 when the display device 100 is rotated 90 degrees. The display device 100 according to the second embodiment has a function to automatically switch the screen display between portrait and landscape orientation depending on the orientation of the display device 118. Therefore, when the orientation of the display device 118 is rotated 90 degrees, the aspect ratio of the display area is reversed. For example, the aspect ratio of the width and height in Figure 6B is 3:2, while the aspect ratio of Figure 6F after 90-degree rotation is 2:3. However, when the aspect ratio is changed in this way, areas (603, 604) where there is no input image occur at the top and bottom of the display area, as shown in Figure 6G.

[0052] On the other hand, Figure 6H shows an example where the input image in Figure 6A is displayed across the entire display area of ​​Figure 6F, even when the aspect ratio is changed by executing the flowcharts shown in Figures 5A and 5B. More specifically, if the aspect ratio of the display area is changed while the display image is being shown, the change is detected by the determination process in S507. Then, S302 to S304 are executed again for the input image in Figure 6A, generating an image with a changed resolution (an example of a "fourth image"), and further generating an extended image (an example of a "second extended image") for this image. Therefore, even if the aspect ratio of the display area is changed, it is possible to display a display image that conforms to the changed aspect ratio.

[0053] The image output unit 203 may also highlight the new image generated by the image expansion unit 202. Highlighting may, for example, enclose the generated image in a frame. Furthermore, highlighting may be applied when the ratio of the area of ​​the newly generated image to the area of ​​the input image exceeds a predetermined threshold. Alternatively, highlighting may be applied when there is a predetermined or greater discrepancy between the aspect ratio of the input image and the aspect ratio of the newly generated image. Alternatively, highlighting may be configured to be arbitrarily specified by the user.

[0054] During the execution of the display processing according to the second embodiment, the display image data output by S305 may be stored in RAM 114 (an example of the "first storage unit"). Under the same conditions, steps S304, S504, and S505 may be omitted, and the display image data stored in RAM 114 may be used. For example, if the user requests the display of an extended image, the image extension unit 202 generates the extended image data shown in Figure 6E in S304. Subsequently, if the display device 118 of the display device 100 rotates 90 degrees during the execution of the same display processing, the image extension unit 202 generates the extended image data shown in Figure 6H in S304. If the display device 118 of the display device 100 rotates 90 degrees again in this state, the extended image data generation process by S304 is not executed. The display image shown in Figure 6E, stored in RAM 114, is displayed again.

[0055] <One aspect of the function and effect> In the second embodiment, the user can arbitrarily select the display method when the aspect ratio of the input image and the display area are different. Furthermore, if the aspect ratio of the resolution of the display area is changed during the display process, a new image is generated to match the changed aspect ratio and the input image is expanded. In this way, even if the aspect ratio of the display area is changed while the display image is being displayed, it is possible to display the image in a way that suppresses a decrease in user satisfaction.

[0056] (Third Embodiment) Next, a third embodiment will be described. The third embodiment relates to the operation of the display device 100 when switching display images. The third embodiment also relates to the operation of the display device 100 when setting an area to expand according to the subject of the input image. In the third embodiment, the configuration of the image generation unit 102 and the processing thereof differ from that of the first embodiment, but the other configurations are the same as those of the first embodiment. Therefore, the same reference numerals are used for components and processes that are the same as or substantially the same as those described above in the first embodiment, and their descriptions are omitted.

[0057] <Example of Functional Configuration of Display Device 100> The functional blocks of the image generation unit 102 in the third embodiment will be explained using Figure 7. The image generation unit 102 is composed of a resolution conversion unit 201, an image output unit 203, a subject detection unit 701, and an image expansion unit 702.

[0058] The subject detection unit 701 performs subject detection on the input image data. The subject detection method involves, for example, inputting the input image data into a known deep learning-based image recognition model to detect subject information. Subject information here refers to, for example, people, animals (dogs, cats, birds, horses, etc.), vehicles (cars, motorcycles, trains, etc.), food, landscapes, or buildings. Furthermore, subject information may be detected using training data that enables detection of individuals or specific entities within people or animals, for example. Note that this machine learning-based subject information detection is just one example, and subject information may also be detected by other methods, such as methods that follow predetermined rules.

[0059] The image expansion unit 702 uses an image generation AI model to generate a new image that is continuous or consistent with respect to the area surrounding the input image output from the resolution conversion unit 201, thereby expanding the input image. At this time, the image expansion unit 702 uses subject information output from the subject detection unit 701 to limit the area to be expanded within the area surrounding the input image. The image generation AI model is the same as that described in the first embodiment. The image output unit 203 outputs the image data output from the resolution conversion unit 201 or the image expansion unit 702 to VRAM.

[0060] <Operation during display processing> The display processing of the display device 100 according to the third embodiment will be explained using Figures 8 and 9A to 9D. In Figure 8, steps that perform the same operations as in the first embodiment are given the same reference numerals and their explanation is omitted. Furthermore, the operation of each step in the flowchart of Figure 8 is realized by the CPU 112 or GPU 116 executing a program stored in the ROM 113 and controlling other hardware of the display device 100 as necessary. In addition, the display operation by the display device 100 is performed in response to user instructions through the input device 117, as in the first embodiment.

[0061] In S801, the image input unit 101 acquires input image data. The image input unit 101 then outputs the acquired input image data and the associated information described above to the image generation unit 102. The image input unit 101 may also process the input image data to make it suitable for use by the image generation unit 102 before outputting it.

[0062] In S802, the resolution conversion unit 201 determines whether the display image data generated from the input image data has already been stored in RAM. If the resolution conversion unit 201 determines that the display image data has been stored, the process proceeds to S305; otherwise, the process proceeds to S302. S302 is the same as in the first embodiment, so its explanation is omitted. After the execution of S302, the process proceeds to S803.

[0063] In S803, the resolution conversion unit 201 determines whether the resolution of the input image acquired by the image input unit 101 is different from the aspect ratio of the display area. If the resolution conversion unit 201 determines that the aspect ratios are different, the process proceeds to S804; otherwise, the process proceeds to S305. In S804, the subject detection unit 701 detects and acquires subject information from the input image data.

[0064] In S805, the image expansion unit 702 sets an area around the input image to expand it by generating a new image, according to the subject information acquired in S804. In the third embodiment, for example, if a person detected as a subject is cut off at the edge of the input image, the expansion area is not set around the person present in the input image, but the expansion area is set around the input image on the opposite side. The method of setting the area is not limited to the method described above, and the area may be set according to the detection result of a subject of a different type than a person. Also, for example, if the input image cannot be expanded in either the up or down direction, the area where the input image does not exist may be blacked out by the process described in the second embodiment.

[0065] The region for generating a new image will be explained using Figures 9A to 9D. Figure 9A illustrates the input image acquired by the image input unit 101 in S801. The aspect ratio of the input image is 16:9. In S804, the subject detection unit 701 detects and acquires subject information such as a dog, a person photographing the dog, and background buildings from the input image of Figure 9A. On the other hand, Figure 9B illustrates the display area of ​​the display device 118, which has an aspect ratio of 3:2.

[0066] Figure 9C illustrates a state in which the input image of Figure 9A is placed in the display area of ​​Figure 9B. When the aspect ratio of the input image and the display area are different in this way, the processing by the resolution conversion unit 201 in S302 results in an area at the edge of the display area where the input image does not exist. In this case, the person in the input image of Figure 9A is cut off at the bottom of the image. Therefore, in S805, the image expansion unit 702 does not expand the image downwards, but positions the input image to align with the bottom edge of the display area so that the image can be expanded upwards. When the input image is positioned in this way, as shown in Figure 9C, an area 901 where the input image does not exist exists on the upper side of the display area. Then, in the processing of S806, a new image is generated for this area 901, and the expanded image is generated.

[0067] More specifically, in S806, the image expansion unit 702 generates a new image with continuity or consistency for the region 901 set in S805, thereby generating a display image. This makes it possible to display the image across the entire display area even when the aspect ratio of the input image and the display area are different. Figure 9D shows an example of how the input image in Figure 9A is expanded and displayed across the entire display area of ​​Figure 9B in this manner. The process then proceeds to S305.

[0068] Since steps S305 and S306 are the same as in the first embodiment, their explanation is omitted. After the execution of S306, the process proceeds to S807. In S807, the display unit 103 determines whether the display state of the image data has been canceled, that is, whether the image data display process has ended. If it is determined that the image data display process has ended, the process ends; otherwise, the process proceeds to S808. The conditions under which the display state of the image data is canceled are, for example, when the display device 118 is turned off, or when the display state of the display device 118 transitions to another mode. Another mode is, for example, a mode in which a list of thumbnail images is displayed, or a mode in which a settings screen for configuring the display device 100 is displayed.

[0069] In S808, the display unit 103 (an example of the "third determination unit") determines whether or not an instruction has been given to display another image while an image is being displayed. If the display unit 103 determines that an instruction has been given to display another image, the process proceeds to S809; otherwise, the process returns to S807. In S809, the image output unit 203 temporarily stores the display image data being displayed in the RAM 114 (an example of the "second storage unit"). Here, the display image data is stored in the RAM 114 solely for display purposes, and the input image data is not stored in the recording medium 115. Also, if the display image data, including the extended portion, is displayed across the entire display unit 103, the update of supplementary information such as image size is not performed. Then, the process returns to S801.

[0070] By performing the steps described above, for example, if an instruction is given to display the input image of Figure 9A, and the display image data for Figure 9D is generated and displayed on the display unit 103, then an instruction is given to display another input image, the following process is executed. That is, the extended image of Figure 9D is temporarily stored in the RAM 114, and the display image data corresponding to the other input image is displayed on the display unit 103. If an instruction is given to display the image of Figure 9A again from that state, the extended image data of Figure 9D stored in the RAM 114 is read out as the display image data and displayed on the display unit 103. In other words, if extended image data is temporarily stored in the RAM 114, this extended image data is displayed, and no new extended image data is generated.

[0071] Furthermore, when a user swipes their finger across the touch panel display unit 103 to display images that are earlier or later than the currently displayed image, the next image that can be displayed is limited. Therefore, between the time the display image data is displayed in S306 and the time the display of another image is instructed in S808, processing equivalent to S302 to S806 may be performed on the next image that can be displayed. In this case, an image with a changed resolution (an example of a "fifth image") is generated, and an extended image (an example of a "third extended image") is generated for this image. Display image data may be generated in advance through such processing and stored in the RAM 114. In this case, when the display of the next image is instructed in S808, processing S809, S801, and S802 are executed in that order, and then processing S302 to S806 can be omitted, and the process can proceed to S305. Thus, the time until the next image is displayed on the display unit 103 can be shortened.

[0072] <One aspect of operation and effect> According to the display device 100 of the third embodiment, when the aspect ratio of the input image and the display area are different, a region 901 is set to expand the image according to the detection result of a subject in the input image. Then, a new image with continuity or consistency is generated for the region 901 to expand the input image. As a result, it becomes possible to display the image across the entire display area regardless of the difference in aspect ratio between the input image and the display area.

[0073] Furthermore, according to the display device 100 of the third embodiment, when the display switches to another image during image display, the display image data being displayed is temporarily stored in the RAM 114. This prevents the expanded image content in area 901 from being different each time it is displayed, even if an image is displayed again after another image has been displayed in a series of display processes, by regenerating the image. Therefore, it becomes possible to display the same image that was initially displayed again after another image.

[0074] (Fourth Embodiment) Next, a fourth embodiment will be described. In the fourth embodiment, when the aspect ratio of the display area and the input image are different, a predetermined condition is used to determine whether or not to generate an image for the blank area of ​​the display area to expand the input image. In the fourth embodiment, the configuration of the image generation unit 102 is the same as in the first embodiment. Therefore, the same reference numbers are used for components and processes that are the same as or substantially the same as those described above in the first embodiment, and their descriptions are omitted.

[0075] <Operation during display processing> The display processing of the display device 100 according to the fourth embodiment will be described using Figures 10 and 11A to 11F. In Figure 10, the same reference numerals are used for steps that perform the same operations as in the first embodiment, and their explanations are omitted. The operation of each step in the flowchart of Figure 10 is realized by the CPU 112 or GPU 116 executing a program stored in the ROM 113 and controlling other hardware of the display device 100 as necessary. The display operation by the display device 100 is performed in response to user instructions via the input device 117, as in the first embodiment.

[0076] Since steps S301 and S302 are the same as in the first embodiment, their explanation is omitted. After the execution of S302, the process proceeds to S1001. In S1001, the resolution conversion unit 201 determines whether the aspect ratio of the input image acquired by the image input unit 101 is different from that of the display area. If the resolution conversion unit 201 determines that the aspect ratios are different, the process proceeds to S1002; otherwise, the process proceeds to S305.

[0077] In S1002, the image input unit 101 determines whether or not the input image data has a generation extension permission flag attached. If the image input unit 101 determines that the generation extension permission flag has been attached, the process proceeds to S1003; otherwise, the process proceeds to S1005. Here, the generation extension permission flag is information indicating whether or not to allow the generation of an image for the blank area that occurs when the aspect ratio of the input image and the display area are different, and to allow the expansion of the input image. Such a generation extension permission flag is embedded in the image file along with the input image data as supplementary information to the input image data.

[0078] In the fourth embodiment, an example of generating an extended image in S1005 when the generation extension permission flag is not set will be described later, but it is not necessary to generate an extended image when the generation extension permission flag is not set. In such a case, the process may proceed to S1004 after S1002 is executed.

[0079] In S1003, the image input unit 101 determines whether the state of the state generation extension permission flag attached to the image is permitted or not. If the image input unit 101 determines that the state is permitted, the process proceeds to S1005; otherwise, the process proceeds to S1004. For example, the input image data is assumed to be an image from a photobook handled as an e-commerce product. In such a case, the publisher sells the product with the generation extension permission flag set to "not permitted" so that the author's and the work's intentions do not change before or after purchase. Depending on the input image data, the setting of the generation extension permission flag may be changed depending on the user's situation, such as before or after purchase.

[0080] In S1004, the image output unit 203 adds a black image to the display area where no input image exists. Figure 11A illustrates an input image with an aspect ratio of 9:16 acquired by the image input unit 101. Figure 11B illustrates the display area of ​​the display device 118 with an aspect ratio of 3:2. Figure 11C illustrates a display image when the vertical width of the input image in Figure 11A is adjusted to match the vertical width of the display area in Figure 11B. In the case of Figure 11C, areas (1101, 1102) where no image exists are created on the left and right sides of the display area, and a black image is added to these areas (1101, 1102).

[0081] In S1005, the image expansion unit 202 does not add a black image to the region (1101, 1102) shown in Figure 11C. That is, the image expansion unit 202 expands the input image by generating a new image for the region (1101, 1102) that is continuous or consistent with the input image. This makes it possible to display the image across the entire display area when the aspect ratio of the input image and the display area are different. Figure 11F illustrates an expanded image in which the input image in Figure 11A is expanded to cover the entire display area of ​​Figure 11B.

[0082] Furthermore, even when the generation and expansion permission flag is enabled, use cases can be considered where the degree of generation and expansion is limited to prevent significant alteration of the input image. In such cases, the display device program may include a threshold that uniformly limits the degree to which the input image can be generated and expanded. Alternatively, a threshold may be set for each input image by adding metadata to the input image data indicating that the expansion rate is limited.

[0083] For example, region 1107 in Figure 11D illustrates the region where the input image, whose resolution has been changed by processing by the resolution conversion unit 201 in S302, is displayed. The expanded regions (1105, 1106) indicate regions where the input image can be expanded. These regions are calculated using a threshold that limits the expansion ratio. In this way, by limiting the expanded regions, regions (1103, 1104) are created where no input image exists around them.

[0084] In such cases, the input image is expanded by generating a new image in the expanded region (1105, 1106) that is continuous or consistent with the input image. Meanwhile, a black image is applied to the region (1103, 1104) where the expanded region (1105, 1106) is not provided. Figure 11E illustrates the display image generated in this way. Note that region (1103, 1104) is an example of a "blank region". Also, region (1105, 1106) is an example of a "predetermined proportion of the blank region". Furthermore, the region (1103, 1104) where the expanded region (1105, 1106) is not provided is an example of a "region other than the predetermined proportion of the blank region".

[0085] Furthermore, the prompt, which is text information, may be embedded in the image file along with the input image data, and an image may be generated in the extended area that represents the content written in the prompt. For example, the extended area may be used as an advertising medium by naturally generating an image of the product logo, sign, or the product itself that is written in the prompt. Alternatively, the input image may be extended by generating a new image with more continuity or consistency by incorporating information about the surrounding circumstances when the image was taken into the prompt. Then the process proceeds to S305. Since S305 and S306 are the same as in the first embodiment, their explanation is omitted. After the execution of S306, the process ends. Note that the prompt is an example of "data attached to the data of the acquired image".

[0086] <One Aspect of Operation and Effect> According to the display device 100 of the fourth embodiment, the control regarding the expansion of the input image is changed according to the information attached to the image. In this way, when expanding the input image and the aspect ratio of the input image and the display area are different, a new image with continuity or consistency with the input image is generated for areas in the display area where the input image does not exist, thereby expanding the input image. This makes it possible to display the image across the entire display area regardless of the difference in aspect ratio between the input image and the display area. Furthermore, according to the display device 100 of the fourth embodiment, conditions are set for when the input image is not expanded. Alternatively, even when expansion is performed, restrictions are placed on the expanded area. In this way, it is possible to protect against alteration of the author's or the artist's intentions, or to expand the input image by generating a new image with more continuity or consistency, as the expanded content is specified by the user.

[0087] (Fifth Embodiment) Next, a fifth embodiment will be described. The contents described in the first embodiment will be omitted from the description. In the fifth embodiment, the display device 100 according to the first embodiment performs the process of generating an extended image when the operation of the display device 100 is set to a predetermined mode. That is, an extended image is not generated when the operation of the display device 100 is not set to a predetermined mode.

[0088] Hereinafter, an editing mode for editing images will be given as an example of a predetermined mode. Figure 12A illustrates the display screen of the editing mode according to the fifth embodiment. This display screen includes a frame 1201 for cropping the subject and an icon 1202 for switching the screen to the playback mode screen. On the other hand, Figure 12B illustrates the display screen of the playback mode according to the fifth embodiment. The playback mode display screen includes an icon 1203 for switching the screen to the editing mode screen.

[0089] The editing mode includes a function to crop images to an arbitrary field of view, and cropping is performed by the user operating frame 1201. Furthermore, the editing mode and playback mode can be switched between by user operation. Specifically, the transition from editing mode to playback mode is performed by operating icon 1202, and conversely, the transition from playback mode to editing mode is achieved by operating icon 1203.

[0090] The display processing of the display device 100 according to the fifth embodiment will be described below with reference to Figures 13 to 15. In the flowcharts of Figures 14 and 15, the same reference numerals are used for steps that perform the same operations as in the first embodiment described above, and their explanations are omitted.

[0091] The overall flowchart of the display process according to the fifth embodiment will be explained using Figure 13. Note that the operation of each step in the flowchart of Figure 13 is realized by the CPU 112 or GPU 116 executing a program stored in ROM 113 and controlling other hardware of the display device 100 as necessary.

[0092] In S1301, the CPU 112 (an example of the "fourth determination unit") determines whether the current operating mode is editing mode. If the CPU 112 determines that the operating mode is editing mode, the process proceeds to S1302; otherwise, the process proceeds to S1303. In S1302, the CPU 112 executes the display process in editing mode. Then the process ends. In S1303, the CPU 112 executes the display process in playback mode. Then the process ends.

[0093] Figure 14 illustrates a flowchart of the detailed operation of the display process in editing mode. Note that the operation of each step in the flowchart of Figure 14 is achieved by the CPU 112 or GPU 116 executing a program stored in ROM 113 and controlling other hardware of the display device 100 as needed.

[0094] In the flowchart shown in Figure 14, the same processing as in the flowchart in Figure 3 according to the first embodiment is performed. That is, a new image with continuity or consistency with the input image is generated for display areas where the input image does not exist, which occurs because the aspect ratio of the input image and the display area are different. Then, the display image expanded by this image is displayed. Figure 12A illustrates the expanded image generated in this way. The editing mode allows cropping within the range including the expanded image generated in this way. Such a display device 100 can increase the degree of freedom of editing. Note that steps S301 to S306 shown in Figure 14 are the same as the steps in the flowchart shown in Figure 3 according to the first embodiment, so their explanation is omitted.

[0095] On the other hand, Figure 15 will be used to illustrate a flowchart of the detailed operation of the display processing in playback mode. Note that the operation of each step in the flowchart of Figure 15 is realized by the CPU 112 or GPU 116 executing a program stored in ROM 113 and controlling other hardware of the display device 100 as needed.

[0096] In the display process shown in Figure 15, a black image is added to the portion of the display area where the input image does not exist, which occurs because the aspect ratio of the input image and the display area are different.

[0097] Steps S301 to S302 are the same as in the first embodiment, so their explanation is omitted. After the execution of S302, the process proceeds to S1501. In S1501, the resolution conversion unit 201 determines whether the aspect ratio of the input image acquired by the image input unit 101 is different from that of the display area. If the resolution conversion unit 201 determines that the aspect ratios are different, the process proceeds to S1502; otherwise, the process proceeds to S305.

[0098] In S1502, the image output unit 203 adds a black image to the areas (401, 402) located above and below the input image shown in Figure 4C. As a result, the display image shown in Figure 12B is displayed. The process then proceeds to S305. Steps S305 to S306 are the same as in the first embodiment, so their explanation is omitted. After the execution of S306, the process ends.

[0099] <One aspect of operation and effect> According to the display device 100 of this fifth embodiment, the processing of the first embodiment is operated when the device is in editing mode. Therefore, in editing mode, it is possible to trim the image to include the generated extended image, thereby increasing the degree of freedom in editing. On the other hand, in playback mode, the user does not intend to generate an extended image, and the process of generating an extended image is not executed. In this way, the process of generating an extended image is limited to editing mode, where the user spontaneously edits the image. Therefore, the unnecessary generation of extended images can be prevented.

[0100] (Sixth Embodiment) Next, a display device 100 according to the sixth embodiment will be described. In the sixth embodiment, when the aspect ratio of the display area is changed during image display, an example of the operation of the display device 100 that performs an animation display that smoothly connects from before to after the aspect ratio change will be given. In the sixth embodiment, the configuration of the image generation unit 102 and the processing thereof differ from that of the first embodiment, but the other processing is the same as in the first embodiment. Therefore, the same reference numerals are used for components and processes that are the same as or substantially the same as those described above in the first embodiment, and their descriptions are omitted.

[0101] <Example of Functional Configuration of Display Device 100> The functional blocks of the image generation unit 102 according to the sixth embodiment will be explained using Figure 16. The image generation unit 102 is composed of a resolution conversion unit 1601, an image expansion unit 1602, an image output unit 1603, and an image storage unit 1604.

[0102] The resolution conversion unit 1601 converts the horizontal or vertical dimensions of the input image to the horizontal or vertical dimensions of the display area while maintaining the aspect ratio of the input image's resolution. Alternatively, it converts the input image in accordance with changes in the display area of ​​the display device 100. Changes in the display area of ​​the display device 100 include, for example, rotating the display device 100 by an arbitrary angle.

[0103] The image expansion unit 1602 uses an image generation AI model to generate a new image that is continuous or consistent with respect to the area surrounding the input image output from the resolution conversion unit 1601, thereby expanding the input image. At this time, the image expansion unit 1602 determines the amount of expansion considering the amount of change in the aspect ratio of the display area. The image generation AI model is the same as that described in the first embodiment.

[0104] The image output unit 1603 outputs the image data output from the resolution conversion unit 1601 or the image extension unit 1602 to the image storage unit 1604 (an example of the "third storage unit"). The image storage unit 1604 then stores this image data. The image storage unit 1604 is, for example, the VRAM of RAM 114.

[0105] <Operation during display processing> An example of the display processing of the display device 100 according to the sixth embodiment will be described using Figures 17 and 18A to 18M. Hereafter, the transition of the image displayed in the display area will be described when the display device 118 is rotated 90 degrees clockwise while the display image is displayed in a horizontally elongated display area as shown in Figure 18C.

[0106] Figure 18A illustrates an input image acquired by the image input unit 101. The aspect ratio of the input image is, for example, 16:9. Figure 18B illustrates the display area of ​​the display device 118. The aspect ratio of the display area is, for example, 3:2. Figure 18C illustrates a display image generated from the input image by the image generation process according to the first embodiment to fit the display area shown in Figure 18B. On the other hand, Figure 18D illustrates the display area when the display device 118 is rotated 90 degrees clockwise.

[0107] The display device 100 according to the sixth embodiment has a function to automatically switch the screen display between portrait and landscape orientation depending on the orientation of the display device 118. Therefore, when the orientation of the display device 118 is rotated by 90 degrees, the aspect ratio of the display area is reversed. For example, the aspect ratio of the width to height in Figure 18B is 3:2, while the aspect ratio of Figure 18D after a 90-degree rotation is 2:3. Note that the display area in Figure 18D is an example of the "third display area".

[0108] Figures 18E to 18H illustrate, as a comparative example, the display transition when the display device 100 is rotated 90 degrees from the display state in Figure 18C, without using the method according to the sixth embodiment. When the display device 100 is rotated from landscape to portrait, the aspect ratio of the display area is inverted as shown in Figure 18E. However, immediately after the inversion, the orientation of the subject included in the display image in Figure 18C does not match the orientation of the inverted display area, as shown in Figure 18E. Subsequently, an animation is displayed to show how to align the orientation of the rotated display area with the orientation of the image. That is, the screen of the display device 118 transitions in the order of Figure 18F and Figure 18G.

[0109] Here, in order to match the horizontal width of the display image with the horizontal width of the display area after the aspect ratio has been changed, the width dimension of the display image is gradually reduced in the order of Figures 18F, 18G, and 18H, starting from Figure 18E. Also, the display image shown in Figure 18E is displayed with a portion that does not fit within the display area cropped out during the transition. When the image transition is complete, the orientation of the image displayed on the screen is finally aligned with the orientation of the display area after the rotation of the subject, as shown in Figure 18H. Finally, the extended generation process according to the second embodiment is executed on the image in Figure 18H to match the aspect ratio after the display area has been changed. Figure 18I shows an example of a display image after such processing has been performed.

[0110] In this comparative example, the image generation process is performed so that the aspect ratio of the input image becomes the aspect ratio of the display area in Figure 18C. Furthermore, after that, an extended generation process is performed after the aspect ratio has been changed, as shown in Figure 18I. However, in this comparative example, during the display of the animation corresponding to the change in the aspect ratio of the display area, areas where there is no display image appear in the corners of the display area. For example, area 1801 in Figure 18F and area 1802 in Figure 18G are examples of areas where there is no display image. Figures 18F and 18G illustrate the display area in a state where such areas 1801 and 1802 are filled in with black.

[0111] Next, a flowchart of the display operation of the display device 100 according to the sixth embodiment will be explained using Figure 17. In Figure 17, steps that perform the same operations as in the first embodiment are given the same reference numerals and their explanations are omitted. Furthermore, the operation of each step in the flowchart of Figure 17 is realized by the CPU 112 or GPU 116 executing a program stored in the ROM 113 and controlling other hardware of the display device 100 as necessary.

[0112] The process in S301 is the same as the process in S301 in Figure 3 of the first embodiment, so its explanation is omitted. The process then proceeds to S1701. In S1701, the resolution conversion unit 1601 of the image generation unit 102 acquires the resolution of the input image and the display area, respectively. The resolution conversion unit 1601 then converts the resolution of the input image data so that the horizontal or vertical width of the input image matches the horizontal or vertical width of the display area, while maintaining the aspect ratio of the input image acquired in S301. The image after the resolution conversion in this way is an example of the "second image".

[0113] In S1702, the image expansion unit 1602 of the image generation unit 102 obtains the resolution of the input image and the display area, and the aspect ratio variation amount of the display area, and determines the expansion amount of the input image. The expansion amount of the input image is determined so that no areas of the display area are not displayed during the animation in which the display image before the aspect ratio of the display area is changed is transitioned to a display image corresponding to the display area after the aspect ratio has been changed.

[0114] The details of S1702 will be explained using Figure 18J. Figure 18J illustrates an extended image obtained by expanding the input image of Figure 18A with the image generation process according to the sixth embodiment. More specifically, the rectangular region 1803 illustrates the region of the input image of Figure 18A. The rectangular region 1804 illustrates the region of the display image generated by executing the image generation process according to the first embodiment. That is, the rectangular region 1804 corresponds to the display region of Figure 18C.

[0115] On the other hand, the rectangular region 1805 illustrates the region of the image generated when the image generation process according to the second embodiment is executed in accordance with the modified display region, assuming that the aspect ratio of the display region has been changed. That is, the rectangular region 1805 corresponds to the display region in Figure 18I. Region 1806 illustrates the circle inscribed in this rectangular region 1805.

[0116] Here, we assume that the input image in Figure 18A is displayed as a display image with its horizontal width matched to the display area in Figure 18B while maintaining its aspect ratio (i.e., Figure 18C), and that the display area is rotated by 90 degrees as shown in Figure 18D. In this case, the image inside area 1806 may be displayed on the screen during animation display. Therefore, in S1702, the amount of expansion of the input image is determined so that the input image can be expanded to at least fill area 1806.

[0117] In the sixth embodiment, an example is given of displaying an animation in which the displayed image rotates and transitions when the display area is rotated. Furthermore, in the sixth embodiment, an example of enlarging or shrinking the image displayed on the screen in such a case is described. In such a case, while maintaining the aspect ratio of the image, it is possible to display the image so that its horizontal or vertical width matches that of the display area after the aspect ratio has been changed.

[0118] The transition image may be displayed without changing its magnification while it is rotating. In this case, the amount of expansion may be determined to be such that the input image is expanded to fill an area that includes at least a circle inscribed in the display area that was displayed in the display area before the change.

[0119] Furthermore, the sixth embodiment illustrates a case where the aspect ratio of the display area is changed by rotation of the display device, but the change in the aspect ratio of the display area may be achieved by arbitrary operation by the user. For example, the image expansion unit 1602 (an example of a "reception unit") may accept user operations to arbitrarily enlarge or reduce the display area using the input device 117. The image expansion unit 1602 may then determine the amount of expansion of the input image as the amount to be expanded assuming that the input image is expanded to the area formed by the maximum horizontal width and maximum vertical width that the display area can take. Moreover, the image expansion unit 1602 may determine the amount of expansion of the input image as the amount that can be expanded to fill the area of ​​a circle inscribed with the longer of the maximum horizontal width and maximum vertical width that the display area can take.

[0120] Returning to the explanation of the flowchart in Figure 17, in S1703, the image expansion unit 1602 generates a new image that is continuous or consistent with the input image using the expansion amount determined in S1702. Then, the image expansion unit 1602 synthesizes the input image data and the newly generated image data to generate data for an expanded image (an example of a "fourth expanded image").

[0121] The image data generated in S1703 only needs to include at least the region using the expansion amount determined in S1702. For example, if the expansion region determined in S1702 is circular, the expanded image may be generated to fill a rectangular region in which that circle is inscribed.

[0122] Furthermore, in consideration of the processing time required for image generation, the image data generated in S1703 may be generated in such a way that the image quality is reduced for areas that are only displayed during the transition display. For example, in the case of Figure 18J, even if the area is inside area 1806, the area outside the rectangular area 1805 is temporarily displayed during the animation display, but is not displayed in the final transition image. Therefore, the image data may be generated in such a way that the image quality of the image displayed in the area outside the rectangular area 1805 (an example of "an area outside the predetermined range") is lower than that of the rectangular area 1805 (an example of "a predetermined range").

[0123] In S1704, the image output unit 1603 outputs the extended image data output by the image extension unit 1602 in S1703 to the image storage unit 1604 (an example of the "third storage unit"). The image output unit 1603 also outputs display image data, which is a cropped portion of the extended image data showing the area to be displayed, to the image storage unit 1604. The image storage unit 1604 then stores this image data.

[0124] In S1705, the display unit 103 reads the display image data stored in the VRAM of the RAM 114 and displays it on the display device 118. In S1706, the display unit 103 determines whether the display state of the display image has been canceled, that is, whether the display process has ended. If the display unit 103 determines that the display process has ended, the process ends; otherwise, the process proceeds to S1707. The conditions under which the display state of the display image is canceled include, for example, when the display device 118 is turned off, or when the display state of the display device 118 transitions to another mode. Another mode may be, for example, a mode in which a list of thumbnail images is displayed, or a mode in which a settings screen for configuring the display device 100 is displayed.

[0125] In S1707, the display unit 103 (an example of the "fifth determination unit") determines whether the aspect ratio of the display area has changed while the display image is being displayed. If the display unit 103 determines that the aspect ratio has changed, the process proceeds to S1708; otherwise, the process returns to S1706.

[0126] In S1708, the resolution conversion unit 1601 acquires the extended image data stored in the image storage unit 1604 in S1704 and converts it into transition image data corresponding to the change in the display area.

[0127] More specifically, the resolution conversion unit 1601 rotates region 1806 in the pre-generated extended image data, as shown in Figure 18J, 90 degrees clockwise to match the display area with a changed aspect ratio. Then, the resolution conversion unit 1601 rotates region 1806 in the opposite direction by a predetermined angle smaller than 90 degrees, for example. The resolution conversion unit 1601 then enlarges the rotated region 1806 while maintaining its aspect ratio. The resolution conversion unit 1601 then crops a transition image including the subject by fitting a frame corresponding to the display area with a changed aspect ratio onto this enlarged region 1806. The transition image thus generated includes a subject facing diagonally. The degree of enlargement of region 1806 may be set to a value such that multiple transition images result in a smooth animation.

[0128] Similarly, if S1708 is executed again, the resolution conversion unit 1601 rotates the region 1806 in the reverse direction by a predetermined angle to generate another transition image. The resolution conversion unit 1601 continues this transition image generation process until the orientation of the subject included in the transition image matches the orientation of the display region with the changed aspect ratio, that is, until the transition image shown in Figure 18M is generated.

[0129] Figures 18K to 18M illustrate transition images sequentially generated in S1708 using the extended image shown in Figure 18J. The transition image shown in Figure 18K has the same rotation angle and scale as the transition image in Figure 18F related to the comparative example. Similarly, Figure 18L corresponds to Figure 18G, and Figure 18M corresponds to Figure 18H. Figure 18M illustrates the final frame among multiple transition images. Thus, when the extended image in Figure 18J is converted into a transition image generated using the method of the sixth embodiment, the blank areas (1801, 1802) present in Figures 18F to 18H do not exist in the transition images from Figure 18K to Figure 18M.

[0130] Note that Figure 18J may also be generated by expanding the display image displayed in the display area after the aspect ratio has been changed, as shown in Figure 18E. In this case, the resolution conversion unit 1601 does not need to rotate the area 1806 90 degrees clockwise to match the display area with the changed aspect ratio. The resolution conversion unit 1601 also reduces the size of the expanded image by a predetermined angle while maintaining its aspect ratio. Then, the resolution conversion unit 1601 crops the transition image including the subject by fitting a frame corresponding to the display area with the changed aspect ratio onto the reduced area.

[0131] Returning to the explanation of the flowchart in Figure 17, in S1709, the image output unit 1603 outputs the transition image data converted in S1708 to the image storage unit 1604 as display image data (an example of "sequential output"). The image storage unit 1604 then stores this display image data. In S1710, the display unit 103 reads the state transition image data (display image data) stored in the image storage unit 1604 and displays it on the display device 118.

[0132] In S1711, the display unit 103 determines whether the image transition has finished. This determination is made, for example, by determining whether the orientation of the subject included in the transition image matches the orientation of the display area whose aspect ratio has been changed. If the display unit 103 determines that the transition has finished, the process proceeds to S1712; otherwise, the process returns to S1708. In S1712, the display unit 103 continues to display the last displayed transition image. The process then returns to S1706.

[0133] In the sixth embodiment, when an animation display is performed in conjunction with a change in the aspect ratio of the display area, the display of the last transition image continues after the animation display ends. However, this does not apply to the display format after the animation display ends. For example, as in the second embodiment, image data generated by performing image generation processing again according to the aspect ratio after the change in the display area may be displayed. In this case, the image exemplified in Figure 18I is used as the image data for display after the animation display ends.

[0134] <One aspect of its function and effect> According to the display device 100 of this sixth embodiment, the input image is generated by extending it beyond the display area, taking into account the amount of change in the aspect ratio of the display area. By doing so, it is possible to suppress the occurrence of blank space in the display area during the animation that transitions the display images. Therefore, it is possible to suppress a decrease in user satisfaction.

[0135] (Modification) In the above embodiment, the generation and display of the extended image are performed in the display device 100, but the generation and display of the extended image may be performed in different devices. That is, for example, the extended image is generated in an imaging device that can be connected to a network. The extended image data is then transmitted via the network to, for example, the user's smartphone (an example of "outputting extended image data"). The extended image is then displayed on the smartphone's display. It is assumed that the aspect ratio information of the display provided in the smartphone has been transmitted to the imaging device in advance. Even with such modifications, the same effects as in the above embodiment can be achieved.

[0136] (Other Embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0137] The technical ideas derived from this disclosure are not limited to the exemplary embodiments disclosed, but are intended to encompass various modifications of the exemplary embodiments, or substitutions with equivalent structures or functions. The scope of the following claims should be interpreted in the broadest way to encompass all such modifications and equivalent structures and functions.

[0138] This application claims priority based on Japanese Patent Application No. 2025-026032, filed on February 20, 2025, and Japanese Patent Application No. 2025-104035, filed on June 19, 2025, and all of the contents of those applications are incorporated herein by reference.

Claims

1. An information processing device comprising: a conversion unit that converts a first image into a second image by maintaining the aspect ratio of the first image and changing the resolution so that the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the display area; a position control unit that controls the position of the second image converted from the first image by the conversion unit in the display area; a generation unit that generates an extended image of the second image by generating a third image in the blank area of ​​the display area that does not overlap with the second image whose position is controlled by the position control unit; and an output unit that outputs the data of the extended image generated by the generation unit.

2. The information processing apparatus according to claim 1, wherein the third image includes an image that is continuous or consistent with the second image, and the continuous or consistent image includes an image inferred using a neural network from the pattern or features of the second image.

3. The information processing apparatus according to claim 1 or 2, wherein the position control unit controls the second image so that, if the second image includes a person, the area around the edge of the second image closest to the person and the edge opposite to that edge become the margin area.

4. The information processing apparatus according to any one of claims 1 to 3, further comprising a first determination unit for determining whether or not the generation of the extended image is permitted, wherein if the first determination unit determines that the generation of the extended image is permitted, the generation unit generates the extended image.

5. The information processing apparatus according to claim 4, wherein the area in which the generation unit generates the third image is a predetermined proportion of the margin area, and when the first determination unit determines that the generation of the extended image is permitted, the generation unit generates the extended image by extending the second image by generating the third image in the predetermined proportion of the area, and the output unit outputs the extended image generated by the generation unit and a black image generated in the area of ​​the margin area other than the predetermined proportion.

6. If the first determination unit determines that the generation of the extended image is not permitted, the output unit outputs the second image and the black image generated for the blank area, as described in claim 4.

7. If the first determination unit determines that the generation of the extended image is not permitted, the output unit outputs an image cropped from the first image in the aspect ratio of the display area, which is the same size as the entire display area, according to claim 4.

8. The system further comprises a first storage unit for storing image data and a second determination unit for determining whether or not the aspect ratio of the display area has been changed. If the second determination unit determines that the aspect ratio of the display area has been changed, the first storage unit stores the data of the extended image in the output state. The conversion unit converts the first image into a fourth image by maintaining the aspect ratio of the first image and changing the resolution so that the vertical or horizontal dimensions of the first image match the vertical or horizontal dimensions of the second display area whose aspect ratio has been changed. The position control unit controls the position of the fourth image in the second display area. The generation unit generates a second extended image by extending the fourth image by generating an image that is continuous or consistent with the fourth image in the blank area of ​​the second display area that does not overlap with the fourth image whose position has been controlled by the position control unit. The output unit outputs the data of the second extended image generated by the generation unit. If the second determination unit determines that the aspect ratio of the display area has been changed to the original aspect ratio, the output unit outputs the data of the extended image stored in the first storage unit, as described in any one of claims 1 to 7.

9. The information processing apparatus according to any one of claims 1 to 8, wherein the output unit highlights and outputs a region in which an image having continuity or consistency with the second image generated by the generation unit is located.

10. An information processing apparatus according to any one of claims 1 to 9, further comprising: a second storage unit for storing image data; a third determination unit for determining whether or not an instruction has been given to display an image other than the extended image in the output state, wherein before the third determination unit makes the determination, the conversion unit converts the other image into a fifth image by changing the resolution; the position control unit controls the position of the fifth image in the display area; the generation unit generates a third extended image by extending the fifth image; if the third determination unit determines that an instruction has been given to display an image other than the extended image, the second storage unit stores the data of the extended image in the output state; the output unit outputs the third extended image generated by the generation unit; and the output unit further outputs the extended image stored in the second storage unit in response to an instruction to display the extended image.

11. The information processing apparatus according to any one of claims 1 to 10, further comprising a fourth determination unit for determining the operating mode of the information processing apparatus, wherein if the fourth determination unit determines that the operating mode is an editing mode for editing image data, the output unit outputs the extended image generated by the generation unit, and if the fourth determination unit determines that the operating mode is a playback mode for reproducing image data, the output unit outputs the second image and a black image generated for the margin area.

12. The information processing apparatus according to any one of claims 1 to 11, wherein the output unit generates an image for the blank area using data attached to the data of the first image.

13. The information processing apparatus according to claim 1, further comprising: a third storage unit for storing image data; a fifth determination unit for determining whether the display area has been changed to a third display area having a different aspect ratio from the display area, wherein, before the determination of the fifth determination unit, the generation unit generates a fourth extended image by generating an image that is continuous or consistent with the second image, thereby extending the second image to a predetermined area larger than the extended image; the third storage unit stores the data of the fourth extended image generated by the generation unit; if the fifth determination unit determines that the display area has been changed to the third display area, the conversion unit converts the data of the fourth extended image stored in the third storage unit into data of a plurality of transition images corresponding to the change from the display area to the third display area; and the output unit sequentially outputs the data of the plurality of transition images.

14. The information processing apparatus according to claim 13, wherein the predetermined area includes an area of ​​such size that no blank area is generated in the third display area when each of the plurality of transition images is superimposed on the third display area.

15. Changing from the display area to the third display area includes maintaining the size of the display area and changing the aspect ratio of the display area, wherein the predetermined area includes at least a circle inscribed in the display area, as described in claim 14.

16. The information processing apparatus according to claim 13, further comprising a reception unit for receiving operations to enlarge or reduce the display area.

17. The information processing apparatus according to claim 16, wherein the predetermined area includes at least the area formed by the maximum horizontal width and maximum vertical width that the display area received by the receiving unit can take.

18. The information processing apparatus according to claim 16, wherein the predetermined area includes at least a circle inscribed within the longer of the maximum horizontal width and maximum vertical width that the display area received by the receiving unit can take.

19. The information processing apparatus according to claim 13, wherein the generation unit generates the fourth extended image by making the quality of the image of a predetermined range within the predetermined region different from the quality of the image outside the predetermined range within the predetermined region.

20. The information processing apparatus according to claim 19, wherein the generation unit generates the fourth extended image, and the quality of the image within the predetermined range is higher than the quality of the image outside the predetermined range, and the predetermined range includes the region displayed in the last transition image among the plurality of transition images.

21. A control method for an information processing device, comprising: a conversion step in which a conversion unit converts a first image into a second image by changing the resolution of the first image so as to match the vertical or horizontal dimensions of the first image to the vertical or horizontal dimensions of the display area; a position control step in which a position control unit controls the position of the second image converted from the first image in the conversion step within the display area; a generation step in which a generation unit generates a third image for a blank area of ​​the display area that does not overlap with the second image whose position has been controlled in the position control step, thereby generating an extended image of the second image; and an output step in which an output unit outputs the data of the extended image generated in the generation step.

22. A program for causing a computer to execute each step in a control method for an information processing device, the control method comprising: a conversion step in which a conversion unit converts a first image to a second image by changing the resolution of the first image so as to maintain the aspect ratio of the first image and to match the vertical or horizontal dimensions of the first image to the vertical or horizontal dimensions of the display area; a position control step in which a position control unit controls the position of the second image converted from the first image in the conversion step within the display area; a generation step in which a generation unit generates a third image for the blank area of ​​the display area that does not overlap with the second image whose position has been controlled in the position control step, thereby generating an extended image of the second image; and an output step in which an output unit outputs the data of the extended image generated in the generation step.

Images