Virtual object display device

Abstract

This virtual object display device comprises: a communication unit that performs communication; a control unit that generates a virtual image object of information acquired by the communication unit; a display unit that displays a background image and the virtual image object; a screen shot acquisition unit that performs first pixel number conversion processing and second pixel number conversion processing to acquire a screen shot; and a saving unit that saves the acquired screen shot.

Description

Virtual Object Display Device

[0001] The present invention relates to a virtual object display device such as a head-mounted display (also referred to as HMD) or AR (Augmented Reality) glasses, which enables screenshot acquisition during content playback for virtual reality (VR, also referred to as Virtual Reality) or mixed reality (MR, also referred to as Mixed Reality).

[0002] A virtual space is set in an HMD, a virtual object is displayed in the virtual space, and users view VR content. In addition, with HMDs or AR glasses, MR content is also viewed by superimposing and displaying virtual objects on real space.

[0003] Furthermore, in personal computers and smartphones, screenshots of display screens are acquired to record and save information obtained from applications or records of application execution (for example, game scores, breakthrough points, etc.), or are used for communication with other people.

[0004] Even when viewing VR content or MR content, recording the execution screen of the VR content or MR content is useful, and it is desirable that HMDs and AR glasses also be provided with a method for acquiring screenshots.

[0005] Japanese Patent Publication No. 2003-109025 describes a method applicable to taking screenshots of HMDs and AR glasses. The method described in JP 2003-109025 involves saving the source information (metadata) that constitutes the screenshot image, making the saved metadata accessible via a network, and facilitating sharing with others. However, the method described in JP 2003-109025 has the drawback that it can be difficult to reproduce the image as it appeared at the time the screenshot was taken. This is because, in a multi-screen system linked with information devices such as personal computers and smartphones, as proposed in Japanese Patent Publication No. 2017-510106, the display data of the information device may be subject to deletion independently of HMD operation, or the display data may contain information that is automatically updated.

[0006] Japanese Patent Publication No. 2003-109025, Japanese Patent Publication No. 2017-510106

[0007] The object of the present invention is to provide a virtual object display device that is equipped with a screenshot acquisition function, and furthermore, a virtual object display device that can acquire screenshots with excellent visibility.

[0008] To solve the above problems, the virtual object display device of the present invention comprises a communication unit that performs communication, a control unit that generates virtual image objects of information acquired by the communication unit, a display unit that displays a background image and the virtual image objects, a screenshot acquisition unit that performs a first pixel count conversion process and a second pixel count conversion process and acquires a screenshot, and a storage unit that stores the acquired screenshot.

[0009] The virtual object display device of the present invention makes it possible to incorporate a screenshot acquisition method even in HMDs and AR glasses. Furthermore, since a screenshot with a higher number of pixels than the display unit can be obtained, and the mirrored image can be viewed as an image with a higher pixel count, less information is lost due to downsizing, and a screenshot that is easy to view can be provided.

[0010] This is a block diagram showing an example of an HMD. This is a block diagram showing an example of an HMD. This is an external view of an example of an HMD. This is a diagram showing an example of an MR space in a multi-display system. This is a diagram showing an example of a screenshot of Figure 4. This is a flowchart showing an example of the processing of a screenshot acquisition program. This is a block diagram showing an example of an HMD. This is a diagram showing an example of an MR space where face detection is performed. This is a diagram showing an example of a screenshot of Figure 8. This is a flowchart showing an example of the processing of a screenshot acquisition program. This is a diagram illustrating an example of a pixel count selection operation for a screenshot. This is a diagram illustrating an example of a partial area selection operation for a screenshot. This is a flowchart showing an example of the processing of a screenshot acquisition program. This is a diagram illustrating an example of a screenshot containing video data. This is a diagram illustrating an example of a screenshot containing wide-area data. This is a flowchart showing an example of the processing of a screenshot acquisition program. This is a diagram illustrating an example of the processing of a screenshot acquisition according to the present invention.

[0011] The embodiments of the present invention will be described below with reference to the drawings.

[0012] [First Embodiment of the Invention] The first embodiment of the invention will be described with reference to Figures 1 to 6. Figures 1 and 2 are block diagrams of the HMD or AR glasses (hereinafter referred to as HMD) of the present invention, and are applicable not only to the first embodiment but also to the second embodiment and subsequent embodiments.

[0013] In the HMD1 shown in Figure 1, reference numeral 10 denotes the camera unit, reference numeral 11 denotes the distance measuring unit, reference numeral 12 denotes the image display unit, reference numeral 13 denotes the acoustic unit, reference numeral 14 denotes the sensor group, reference numeral 15 denotes the control unit, reference numeral 16 denotes the internal bus. Furthermore, within the control unit 15, reference numeral 20 denotes the communication unit, reference numeral 21 denotes the main control unit, reference numeral 22 denotes the screenshot acquisition unit, reference numeral 23 denotes the determination unit, reference numeral 24 denotes the first pixel count conversion unit, reference numeral 25 denotes the second pixel count conversion unit, and reference numeral 26 denotes the screenshot storage unit, which is the area where the acquired screenshot is stored.

[0014] When a user wears the HMD1 and views VR or MR (also known as CGR: Computer Generated Reality) content, the main control unit 21 manages the CGR content playback process and the overall operation of the HMD1. The camera unit 10 captures the real space in front of the HMD1. If the camera unit 10 is equipped with multiple cameras, or if it is a 360-degree camera, it may capture the 360-degree surroundings of the HMD1. The main control unit 21 detects real objects such as furniture and people from the images captured by the camera 10. The distance measuring unit 11 obtains distance information to the detected real objects and also renders virtual image objects. The image display unit 12 displays the CGR background image and virtual image objects, and the sound unit 13 outputs the CGR sound objects. The CGR's acoustic objects may be three-dimensional acoustic objects, and after being processed for three-dimensionalization by the main control unit 21, they are output to an external sound amplification device such as headphones via the acoustic unit 13. The sensor group 14 consists of an acceleration sensor, a gyroscope, a geomagnetic sensor, a GPS sensor, etc., and perceives, for example, the user's up, down, forward, backward, left, and right movements, as well as rotational movements accompanied by changes in the direction of their gaze.

[0015] Furthermore, the camera unit 10 may be equipped with infrared illumination to improve infrared light sensitivity, taking into consideration shooting in dark areas. In addition, the distance measuring unit 11 may be a LiDAR or other device that measures the real space in three dimensions using laser light to improve detection accuracy.

[0016] Furthermore, the communication unit 20 is equipped with several communication protocols, including Internet communication, mobile communication, and a USB interface, and is used depending on the environment. In Figure 1, the HMD 1 is connected to a personal computer (PC) 100 and a smartphone 101, which are cooperating information devices, via the communication unit 20.

[0017] In HMD1 in Figure 2, the same blocks as those in HMD in Figure 1 are assigned the same numbers, and redundant explanations are omitted. In HMD1 in Figure 2, the control unit 15 consists of a main calculation unit 30, a memory unit 31, a communication unit 20, and a storage unit 32, with reference numeral 33a being the CGR playback program and reference numeral 34 being the screenshot acquisition program.

[0018] The main processing unit 30 consists of a CPU, etc., and the memory unit 31 consists of RAM, etc., and constitutes a computer system. The storage unit 23 consists of a non-volatile storage medium such as flash ROM and stores basic operating programs such as the operating system 33, a CGR playback program 33a, a screenshot acquisition program 34, and a screenshot saving unit 26, which are not shown in this figure. The CGR playback program 33a and the screenshot acquisition program 34 are loaded into the memory unit 31 and executed by the main processing unit 30. In Figure 2, the CGR playback program 33a and the screenshot acquisition program 34 are shown as separate processes, but this is not limited to this, and a single program may include both the CGR playback function and the screenshot acquisition function, or they may be programs in different partitioned forms.

[0019] Furthermore, the calculation processing of the main calculation unit 30 may be partially or entirely performed by a server on the Internet via the communication unit 20, or by a personal computer connected via direct communication.

[0020] Figure 3 is an external view of HMD1, and the numbers correspond to the block diagrams of the HMD in Figures 1 and 2.

[0021] In Figure 3, reference numeral 10 denotes the camera unit, and reference numeral 11 denotes the distance measuring unit for measuring the distance to a real object. The image display unit 12 is equipped with display elements for the left eye and the right eye, and displays the left eye image and the right eye image, allowing the user to recognize a three-dimensional image. Note that the image display unit 12 may be of any other type as long as it is capable of three-dimensional display.

[0022] Reference numeral 15 denotes the control unit, reference numerals 13a and 13b denotes the speaker, reference numeral 13c denotes the microphone (corresponding to reference numeral 13 in Figure 1), reference numeral 14 denotes the sensor group, and reference numerals 17a and 17b denotes the mounting housing.

[0023] Figure 4 is a diagram illustrating the relationship between the HMD user and the HMD display unit. The real-world space is, for example, a server room, and the HMD user is a maintenance worker.

[0024] In Figure 4, reference numeral 1 denotes the HMD, reference numeral 2 denotes the HMD user, reference numeral 41 denotes the HMD display image, reference numeral 42 denotes the background image, reference numeral 43 denotes the PC's virtual image object, reference numeral 44 denotes the PC's mirrored image, reference numeral 45 denotes the smartphone's virtual image object, and reference numeral 46 denotes the smartphone's mirrored image. Mirroring is a function that displays the screen of a smartphone or PC on a television or large display, etc. By connecting the device and the display via Wi-Fi, etc., it becomes possible to display the device's screen in full screen on a large display. Here, based on this principle, we assume a case where a multi-screen display is realized that displays the screens of multiple devices simultaneously by adjusting the size of the screen of each device. In MR, the background image 42 is the real space captured by the camera, and the HMD display image 41 has virtual image objects (43, 45) of information devices superimposed on the background image 42, with mirrored images (44, 46) superimposed.

[0025] User 2 of the HMD is, for example, a server maintenance worker, and is prompted to act according to the instruction screen displayed on the mirrored image 44 of the PC. The mirrored image 46 of the smartphone records the completed maintenance work along with the user ID of user 2 of the HMD. For example, user 2 of the HMD speaks each time a maintenance item is completed, and the smartphone recognizes this and adds it to the displayed data. This automatically creates a maintenance record.

[0026] In VR, the background image is also a virtual image object, and the server displayed in the background image, along with mirrored images of the PC and smartphone (44, 46), are used to train the HMD user 2 in server maintenance work.

[0027] Figure 5 shows (a) a screenshot 200 with a normal resolution, which is a screenshot of the HMD display image as is, and (b) a screenshot 201 with an increased resolution.

[0028] In a standard-resolution screenshot (200), a mirrored image (44 and 46 in Figure 4) is displayed on a portion of the HMD1's display area. However, due to limitations in the number of displayable pixels, it is not possible to display all of the information device's display data. The number of pixels displayed in the mirrored image is downsized from the display data, depending on the size of the information device's virtual image object. As a result, the mirrored image in screenshot 200 has lower image resolution, which may result in parts that are difficult to see.

[0029] On the other hand, when Screenshot 201 obtains a mirrored image from the display data of the information device, it generates a screenshot mirrored image with a higher pixel count than the display mirrored image of Screenshot 200 (first pixel count processing). As a result, the mirrored image of Screenshot 201 is larger than the mirrored image of Screenshot 200, assuming the same pixel density. To accommodate the larger mirrored image, the size of the virtual image object of the information device is also increased, and furthermore, the background image is also enlarged (second pixel count processing). The enlargement of the virtual image object and background image is generated from the virtual image object and background image of Screenshot 200 by processing such as pixel interpolation.

[0030] Figure 6 is the first flowchart of the screenshot acquisition program 34 shown in Figure 2. The screenshot acquisition program 34 is loaded into the memory unit 31 of the HMD1 and executed by the main calculation unit 30.

[0031] Processing starts in S10. Camera capture and distance measurement are performed in S11. A background image is generated in S12. In the case of MR, it is generated from the camera capture image, and in the case of VR, it is generated as one of the virtual image objects. In S13, virtual image objects such as information devices are generated.

[0032] In S14, display data from a connected information device is received. In S15, a mirrored image is generated. Specifically, the steps of determining the ratio of the size of the virtual image object of the information device to the size of the HMD's display unit, obtaining the number of pixels of a mirrored image that can be superimposed by multiplying the number of pixels of the display unit by the size ratio, and generating a mirrored image with a number of pixels that can be superimposed from the received display data of the information device are performed. In S16, the background image, virtual object image, and display mirrored image are displayed on the HMD's display unit.

[0033] Steps S11 to S16 are common to the CGR playback program 33a, and the results executed by the CGR playback program 33a may be used in the screenshot acquisition program 34.

[0034] In S17, it is checked whether a request for screenshot acquisition has been issued from the control unit used by the user. The screenshot is taken by the control unit upon detection of the user's instruction, and all or part of the image display unit 12 can be specified. If a request is made, the pixel ratio is determined in S18. The pixel ratio is the ratio of the number of pixels in the display mirrored image to the display data received from the information device. A small pixel ratio indicates that the display mirrored image has been heavily downsized, and there is a high possibility that there are issues with the visibility of the display mirrored image. For this reason, the first pixel processing in S19 and the second pixel processing in S20 are used to generate a screenshot with an increased number of pixels.

[0035] In S19, if the pixel ratio is smaller than a predetermined value, the predetermined value becomes the pixel conversion ratio for generating a mirrored image for screenshots with a higher pixel count from the display data of the information device. In other words, the size of the mirrored image for screenshots is determined by the predetermined value. For example, if the number of pixels in the display data of the information device is 200 and the number of pixels in the mirrored image for display is 25, the pixel ratio becomes 1 / 8. If the predetermined value is 1 / 4, the number of pixels in the mirrored image for screenshots becomes 50, obtained by converting the pixel ratio of the display data of the information device to 1 / 4, and the mirrored image is enlarged by two times. These pixel counts are for the horizontal direction; if the number of pixels in the vertical direction is also considered, the total number of pixels will be enlarged by four times. The predetermined value may be settable by the HMD user.

[0036] In the pixel ratio determination in S18, if it is greater than a predetermined value, the display mirrored image is used for the screenshot.

[0037] In S20, the number of pixels in the virtual object image and background image of the information device is increased to match the number of pixels in the mirrored image for the screenshot. If the mirrored image is enlarged by a factor of two, the number of pixels in the virtual object image and background image displayed on the HMD's display is also doubled. Pixel interpolation may be used as the method for converting the number of pixels. Note that if horizontal and vertical orientations are taken into consideration, the total number of pixels will be quadrupled.

[0038] In S21, if the pixel ratio in S18 is large, the HMD display image, including the display mirroring image, is saved as a screenshot. If the pixel ratio in S18 is small, a screenshot with increased pixel count generated in S19 and S20 is saved. It is reasonable to use encoding processing that can compress image redundancy when saving. If the pixel count of the background image, etc., is increased through interpolation processing, only redundancy is added, and by performing encoding processing, the increase in the size of the saved screenshot can be suppressed.

[0039] If no screenshot request is made in S17, or after saving a screenshot in S21, the program checks the termination conditions in S22. If the termination conditions are not met and the program continues, it returns to the point before S11. If the termination conditions are met, it terminates in S23.

[0040] As described above, the virtual object display device and the screenshot acquisition method for the virtual object display device according to the first embodiment of the present invention make it possible to equip HMDs and AR glasses with a screenshot acquisition method, and furthermore, it is possible to obtain a screenshot with a larger number of pixels than the display unit, and the mirrored image can be viewed as an image with a larger number of pixels, so that less information is lost due to downsizing and a screenshot that is easy to view can be provided.

[0041] [Second Embodiment of the Invention] A second embodiment of the invention will be described with reference to Figures 7 to 10. In this embodiment, the example described is the acquisition of a screenshot suitable for recognizing a person's face, but the invention can also be applied to examples of acquiring screenshots suitable for recognizing characteristic parts of people or other objects other than faces.

[0042] Figure 7 shows the HMD of the present invention, which is identical to HMD1 in Figure 2. The difference lies in the device connected to HMD1, which is connected to a server 102 on the cloud.

[0043] Figure 8 is a diagram illustrating the relationship between the HMD user and the HMD display unit. The real-world space is, for example, a conference room, and the HMD user is giving a presentation. In Figure 8, components identical to those shown in Figure 4 are given the same numbers.

[0044] In FIG. 8, reference numeral 42 denotes a background image of a real space obtained by capturing an image of a conference room with a camera; reference numerals 50 and 51 denote virtual image objects of remote participants participating in the conference via a network; reference numerals 52 to 55 denote images obtained by capturing images of conference participants in the real space with a camera; and reference numeral 56 denotes a virtual image object for showing an outline of a presentation to an HMD user 2 and assisting the user in performing the presentation.

[0045] The HMD user 2 operates the HMD 1 to acquire a screenshot, for example, for the purpose of recording conference participants.

[0046] FIG. 9 shows (a) a screenshot image 202 with a normal number of pixels obtained by directly screenshotting a display image of an HMD, and (b) a screenshot 203 with an increased number of pixels.

[0047] The screenshot 202 with a normal number of pixels is a screenshot of a display image of the HMD 1. When there are many participants and the number of pixels allocated for displaying the participants is small, it may become difficult to identify and visually recognize the participants.

[0048] On the other hand, for the screenshot 203, face images of participants are detected (the detected face images are enclosed by square broken lines), and the number of pixels of the face images is increased (a first pixel number processing described later). As a result, when the pixel density is the same, the face image in the screenshot 203 is larger than the face image in the screenshot 201. In order to accommodate the enlarged face image, the number of pixels of the background image is also increased and enlarged (a second pixel number processing).

[0049] FIG. 10 is a second flowchart of the screenshot acquisition program 34 shown in FIGS. 2 and 7. The screenshot acquisition program 34 is expanded in the memory unit 31 of the HMD 1 and executed by the main calculation unit 30. In addition, the same steps as those in the first flowchart described with reference to FIG. 6 are assigned the same reference numerals, and the description thereof may be omitted.

[0050] Processing starts in S10. Camera capture and distance measurement are performed in S11. A background image is generated in S12. In the case of MR, it is generated from the camera capture image, and in the case of VR, it is generated as one of the virtual image objects. In S30, the virtual image object of the remote participant is received from server 102. In S31, the background image and virtual image object are displayed.

[0051] Steps S11-S12, S30, and S31 are common steps with the CGR playback program 33a, and the results executed by the CGR playback program 33a may be used in the screenshot acquisition program 34.

[0052] In S17, it is checked whether a request to take a screenshot has been issued from the control unit used by the user. If a request is made, in S32, the face images of the remote participant and the participant in the real world are detected from the HMD display image. In S33, the number of pixels in the face image is determined. There is a minimum number of pixels N (which may be set by the HMD user) that is considered necessary to identify a person from the face image. In the face pixel count determination S33, if the smallest number of pixels M (per area) in the face image detected in S32 is less than the number of pixels N (per area), a value exceeding N / M is set as the parameter L for increasing the number of pixels in the screenshot. However, a lower limit value for M may be set to restrict the pixel increase parameter L from becoming excessively large, so that the screenshot does not become too large.

[0053] Steps S34 to S36 constitute the first pixel count conversion process. In S34, a request is made to the server 102 to resend a virtual image object of the face image having N pixels, and it is received. Steps S35 and S36 are steps to multiply the number of pixels of the face image captured by the camera by L. In S35, the zoom ratio of the camera is temporarily set to √L, and the camera is panned and tilted while searching for and capturing a face image, and the captured face image is saved. S21 is the second image conversion process step, in which the background image is multiplied by √L both horizontally and vertically to match the face image with increased pixel count. Note that for a brief moment while the camera is panning and tilting, the background image may be fixed to the previous image.

[0054] In S21, if the number of face pixels in S33 is large enough (OK), the HMD display image is saved as a screenshot. If the number of face pixels in S33 is small, a screenshot with increased pixel count generated in S34 to S36 is saved.

[0055] As described above, the virtual object display device and the screenshot acquisition method for the virtual object display device according to the second embodiment of the present invention have the same features as the first embodiment, and also have the feature that screenshots can be acquired in which characteristic parts of a person or object, such as a face, can be easily identified.

[0056] [Third Embodiment of the Invention] A third embodiment of the invention will be described with reference to Figures 11 to 13. Figure 11 is a diagram showing the relationship between the HMD user and the HMD display unit. Similar to Figure 4, the real space is a space such as a server room, and the HMD user is a maintenance worker. In Figure 11, the same components as those shown in Figure 4 are given the same numbers.

[0057] In Figure 11, a standard pixel count button object 60 and a high pixel count button object 61 are added to the HMD display image 41. The HMD user 2 can, for example, select one of the buttons by moving their hand to fix the pixel count of the screenshot.

[0058] When the standard pixel count button object 60 is pressed, a screenshot with the normal pixel count is taken regardless of the determination result in S18 in Figure 13. When the high pixel count button object 61 is pressed, a screenshot with the normal pixel count or a screenshot with increased pixel count is taken according to the determination result in S18 in Figure 13.

[0059] Figure 12 is a diagram illustrating the relationship between the HMD user and the HMD display unit. Similar to Figure 4, the real-world space is, for example, a server room, and the HMD user is a maintenance worker. In Figure 12, the same components as those shown in Figure 4 are assigned the same numbers.

[0060] In Figure 12, when HMD user 2 draws a frame 63 near HMD user 2 by moving their finger 62, a screenshot of a portion area 64 of the HMD display image 41 on the extension of the frame 63 is taken.

[0061] Figure 13 is a third flowchart of the screenshot acquisition program 34 shown in Figures 2 and 7. The screenshot acquisition program 34 is loaded into the memory unit 31 of the HMD1 and executed by the main calculation unit 30. Also, the same numbers are assigned to the same steps as in the first flowchart explained in Figure 6, and explanations may be omitted in some cases.

[0062] The third flowchart in Figure 13 adds steps S40 and S41 to the first flowchart in Figure 6. S40 is the process for selecting the number of pixels for the screenshot shown in Figure 11. For example, if the standard pixel count button object 60 is pressed, the result of the pixel ratio determination step in S18 is fixed to a result with a large pixel count ratio, and a normal screenshot is saved in S21. If the high pixel count button object 61 is pressed, a screenshot with a normal number of pixels or a screenshot with increased pixels is selected according to the result of the pixel ratio determination step in S18. S41 is the process for taking a screenshot of a partial area shown in Figure 12.

[0063] As described above, the virtual object display device and the method for acquiring screenshots of the virtual object display device according to the third embodiment of the present invention have the same features as the first and second embodiments, and also have the feature of increasing the degree of freedom in acquiring screenshots.

[0064] [Fourth Embodiment of the Invention] A fourth embodiment of the invention will be described with reference to Figures 14 to 16. Figure 14 is a diagram showing the relationship between the HMD user and the HMD display unit. Similar to Figure 4, the real space is a space such as a server room, and the HMD user is a maintenance worker. In Figure 14, the same components as those shown in the diagram showing the relationship between the HMD user and the HMD display unit in Figure 4 are assigned the same numbers.

[0065] Figure 14 shows the case where the mirrored image 44 contains a portion of the video data 70. In this case, the original video data 71 of the portion of the video data 70 is saved as accompanying data to the screenshot.

[0066] Figure 15 shows the case where the mirrored image 44 includes a portion 72 of the wide-area data. In this case, the original wide-area data 73 of the portion 72 is saved as accompanying data to the screenshot. That is, if the image of the original data (original wide-area data 73) is partial in the mirrored image 44, the entire image of the original data (original wide-area data 73) is saved as accompanying data to the screenshot.

[0067] Figure 16 is the fourth flowchart of the screenshot acquisition program 34 shown in Figures 2 and 7. The screenshot acquisition program 34 is loaded into the memory unit 31 of the HMD1 and executed by the main calculation unit 30. Also, the same numbers are assigned to the same steps as in the first flowchart explained in Figure 6, and explanations may be omitted in some cases.

[0068] The fourth flowchart in Figure 16 adds step S50 to the first flowchart in Figure 6. Step S50 is the step of determining the attributes of the data that constitute the mirrored images shown in Figures 14 and 15, and saving the video data and wide-area data as accompanying data to the screenshot.

[0069] As described above, the virtual object display device and the screenshot acquisition method for the virtual object display device according to the fourth embodiment of the present invention have the same features as the first embodiment, and also have the feature that detailed video data, wide-area data, etc. can be saved along with the screenshot image.

[0070] The embodiments of the present invention described above in Figures 1 to 16 are not limited to these, and it is possible to replace a part of the configuration of one embodiment with that of another embodiment. Furthermore, it is possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention, and the numbers, messages, etc. that appear in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0071] Furthermore, the functions of the invention may be implemented in hardware, either partially or entirely, by designing them as an integrated circuit, for example. Alternatively, they may be implemented in software by a microprocessor unit, CPU, etc., interpreting and executing an operating program. Moreover, the scope of software implementation is not limited, and hardware and software may be used in combination.

[0072] Here, let's reiterate the differences based on the type of virtual object display device being considered, as well as the resolution of screenshots, particularly the resolution of background images.

[0073] Virtual object display devices are broadly classified into immersive HMDs (Head-Mounted Displays) and AR glasses. In immersive HMDs, the external world is displayed by projecting images captured by the rear camera onto the display. Therefore, during the projection process on the HMD, the resolution of the external world may be lower than the resolution of the camera. In contrast, with AR glasses, the external world is observed with the naked eye through the glass, but when taking screenshots, images captured by the rear camera are used, similar to HMDs.

[0074] Therefore, when taking high-resolution screenshots, immersive HMDs offer the option to enhance the resolution of not only mirrored AR objects but also the surrounding environment. In this case, the maximum resolution would be the resolution of the rear camera. On the other hand, in the case of AR glasses, there is no need to reduce the resolution of the rear camera, so the resolution of the surrounding environment defaults to the resolution of the rear camera. Immersive HMDs include, for example, those that, unlike optical see-through types, completely block the user's view from the outside world and display images captured by the camera on a screen (sometimes called VR goggles).

[0075] Furthermore, the screenshot extraction process according to the present invention will be summarized again with reference to Figure 17. As shown in Figure 17(a), in screenshot extraction, the resolution of the mirroring source image is reduced to a resolution corresponding to the display image of the virtual object display device, and AR display data of this resolution is generated. The virtual object display device also acquires the video from the rear camera as the background image. The generated AR display data and the video data from the rear camera are then combined to generate display data for the virtual object display device, and the virtual object display device performs display based on this display data. Here, as an example, when a screenshot extraction instruction is given, the display data stored in memory is directly extracted to acquire the screenshot.

[0076] In contrast, as shown in Figure 17(b), in this invention, at the moment a screenshot extraction command is issued, for example, the image related to the mirroring source image is adjusted to the resolution within the screenshot, and the adjusted data is stored in memory. Also, background image data for the screenshot is acquired from the rear camera's video feed, and the acquired data is stored in memory. The background image may be adjusted depending on the case. Then, the background image and the adjusted mirroring source image are combined to acquire the screenshot. This makes it possible to acquire a screenshot with good visibility.

[0077] According to the technology of the present invention, for example, in situations where the situation is changing moment by moment (for example, when a presentation is being given), it becomes possible to take a screenshot with good visibility using a virtual object display device. The acquired screenshot can then be saved and used later.

[0078] To solve the above-mentioned problems, as an example, the virtual object display device of the present invention consists of a camera unit, a distance measuring unit, a display unit, a communication unit, and a control unit. The camera unit captures images of the area in front of the user of the virtual object display device, or the entire 360-degree area, and detects the background of the real space and real objects. The distance measuring unit obtains the distance between the user and the real object. The display unit displays virtual image objects of VR / MR content in three dimensions. The communication unit includes internet communication, mobile communication, and several USB interfaces.

[0079] The control unit includes a computer system consisting of a CPU and memory, and is responsible for the overall operation of the VR content viewing device. It also includes a judgment process, a first pixel count conversion process, and a second pixel count conversion process.

[0080] The control unit obtains display data from information devices such as personal computers and smartphones that are connected via the communication unit. It also generates virtual image objects of the information devices and sends them to the display unit. Multiple information devices may be connected. In MR content, the display unit overlays virtual image objects of information devices onto the background image of the real space captured by the camera unit, and at this time, it uses the depth information of the distance measuring unit to position the virtual image objects of the information devices three-dimensionally. In VR content, virtual image objects of information devices are positioned three-dimensionally on the virtual background image of the virtual space.

[0081] The mirrored image of the information device, which is superimposed (mirrored) onto the virtual image object of the information device, is downsized to a number of pixels that can be superimposed onto the virtual image object of the information device, based on the number of displayable pixels of the display unit of the virtual object display device and the size of the virtual image object.

[0082] Screenshots are taken by the control unit upon detection of user instructions, and the user can specify all or part of the display area of ​​the virtual object display device. Furthermore, the number of pixels in the screenshot can exceed the number of pixels of the virtual object display device (or the number of pixels corresponding to the portion being screenshotted). For this purpose, the control unit's first and second pixel count conversion processes are used.

[0083] In the first pixel count conversion process, a screenshot mirror image is generated that has more pixels than the downsized display mirror image. In the second pixel count conversion process, the background image is converted to a background image with more pixels than the display area of ​​the virtual object display device, maintaining size consistency with the screenshot mirror image.

[0084] As an example, the screenshot acquisition method for the virtual object display device of the present invention generates a screenshot mirror image having a larger number of pixels than the downsized display mirror image in the first pixel count conversion process, based on the number of pixels in the display area of ​​the virtual object display device and the size of the virtual image object. Furthermore, in the second pixel count conversion process, the background image is converted to a background image with a larger number of pixels than the display area of ​​the virtual object display device, maintaining consistency in size with the screenshot mirror image.

[0085] The HMD functions described in this specification (Figures 1, 2, and 7) can also be implemented through integration with smartphones or other devices to reduce the weight of the main unit and extend battery life. Furthermore, while mirroring usually only affects the screen portion, in this specification (Figures 4, 5, 11, 12, 14, and 15), the display includes an outline of the smartphone or PC to identify the type of source device.

[0086] 1: HMD, 10: Camera unit, 12: Image display unit, 15: Control unit, 20: Communication unit, 22: Screenshot acquisition unit, 23: Judgment unit, 24: First pixel count conversion unit, 25: Second pixel count conversion unit, 26: Screenshot saving unit, 30: Main calculation unit, 31: Memory, 32: Storage unit, 33a: CGR playback program, 34: Screenshot acquisition program. 2: HMD user, 41: HMD display image, 42: Background image, 43, 45: Virtual image object of information device, 44, 46: Mirrored image, 200: Screenshot with normal pixel count, 201: Screenshot with increased pixel count. 50, 51: Virtual image object of remote participant, 52-56: Image taken of participant in real space, 202: Screenshot with normal pixel count, 203: Screenshot with increased pixel count. 60, 61: Screenshot pixel count selection button; 62, 63: HMD user area selection operation; 64: Partial area of ​​the HMD display image; 70, 71: Video data; 72, 73: Wide-area data.

Claims

1. A virtual object display device comprising: a communication unit for performing communication; a control unit for generating virtual image objects of information acquired by the communication unit; a display unit for displaying a background image and the virtual image objects; a screenshot acquisition unit for performing a first pixel count conversion process and a second pixel count conversion process and acquiring a screenshot; and a storage unit for saving the acquired screenshot.

2. A virtual object display device according to claim 1, wherein the communication unit connects to an information device and acquires display data from the information device; the control unit acquires a display mirroring image obtained by downsizing the display data; the display unit displays a virtual image object of the information device by superimposing a background image and the display mirroring image; the screenshot acquisition unit, in the first pixel count conversion process, creates a screenshot mirroring image with a higher pixel count than the display mirroring image from the display data of the information device; the screenshot acquisition unit, in the second pixel count conversion process, acquires a screenshot background image with increased pixels at the ratio of the pixel count of the screenshot mirroring image to the pixel count of the display mirroring image; and the screenshot acquisition unit acquires an image obtained by superimposing the screenshot background image and the screenshot mirroring image as the screenshot.

3. A virtual object display device according to claim 2, wherein, when the downsizing ratio of the display mirroring image is greater than a predetermined value, the screenshot acquisition unit acquires an image obtained by superimposing the background image displayed by the display unit and the display mirroring image as the screenshot.

4. A virtual object display device according to claim 2, wherein, when the downsizing ratio of the display mirroring image is smaller than a predetermined value, the screenshot acquisition unit generates the screenshot mirroring image such that the pixel ratio with the display data of the information device is a predetermined value.

5. A virtual object display device according to claim 1, wherein the virtual object display device detects a person or a specific part of an object from a background image and a virtual image object, and the screenshot acquisition unit performs a determination process to compare the number of pixels of the detected specific part with a predetermined number, and when the number of pixels of the detected specific part is less than the predetermined number, in the first pixel count conversion process, increases the number of pixels of the specific part by the ratio of the predetermined number / the number of pixels of the detected specific part, in the second pixel count conversion process, acquires a background image with increased pixels by the ratio of the predetermined number / the number of pixels of the detected specific part, and acquires the background image with increased pixels and the image of the specific part as a screenshot.

6. A virtual object display device according to claim 5, characterized in that, if the number of pixels of a specific part is greater than a predetermined number in the determination process, the screenshot acquisition unit acquires the display image of the display unit as a screenshot with a normal number of pixels.

7. A virtual object display device according to claim 1, characterized in that it comprises an operating unit used for selecting the number of pixels for a screenshot and / or for taking a screenshot of at least a portion of the display area of ​​a displayed image.

8. A virtual object display device according to claim 2, wherein the display data of the information device consists of several partial data, and the virtual object display device acquires video data and / or the original data of the display mirroring image from the partial data in association with a screenshot.

9. A virtual object display device according to claim 1, wherein the virtual object display device is an AR glasses.

10. A virtual object display device according to claim 1, wherein the virtual object display device is a VR goggle.

Images