Aerial display system and aerial image formation method

Abstract

This aerial display system (100) comprises a display device (11) that displays an image, an aerial image formation member (12) that refracts first image light incident from the display device (11) and forms an image in the air, a wall (20) in which there is formed an opening (21) through which at least some of the first image light passes, and a projection device (30) that projects second image light onto a portion of the wall (20) other than the opening. The projection device (30) is disposed on the same side as the display device (11) and the aerial image formation member (12) with the wall as a boundary, and at least some of the portion of the wall (20) onto which the second image light is projected is configured to at least partially transmit the second image light, thereby expanding a visual effect imparted to an observer.

Description

Air Display System and Air Imaging Method

[0001] The present invention relates to a system and method for forming a real image of a video displayed by a display device in the air.

[0002] Conventionally, an optical element for forming a real image of a video displayed by a display device in the air has been known. As an example of such an air imaging optical element, for example, Patent Document 1 discloses an optical element of a two-plane orthogonal reflector type. In this method, by using two arrays of reflecting surfaces arranged orthogonally and sequentially reflecting the incident video light in directions orthogonal to each other, a real image of the video can be formed in the air. By using such an air imaging optical element, it is said that a bright and clear real image can be obtained with high light utilization efficiency.

[0003] Patent No. 5856357

[0004] By the way, the air imaging optical element of Patent Document 1 is assumed to be used for a non-contact input device, and a real image of an icon image is formed in the space near the user's hand, and the user can detect that the real image has been indicated by a finger or a touch pen, etc., so that signal input can be performed without physically contacting the display device. In this case, since the user only needs to look in a relatively close range near the hand, it is easy to focus on the imaging surface of the image, and as a result, the image is easily and clearly visually recognized by the user.

[0005] On the other hand, considering applying such an air imaging optical element to a relatively large video work or a signage system installed outdoors or in a storefront, etc., it is necessary to form a real image of the video in a relatively large exhibition space. However, the conventional air imaging optical element only focuses on allowing an observer to visually recognize an air image (such as an icon image, etc.), and has not been studied on allowing an observer to visually recognize other videos and the air image at the same time. For this reason, the visual effect given to the observer by the conventional air imaging optical element has been limited.

[0006] Furthermore, in the case of relatively large video works, in order for viewers to clearly perceive the projected image in space, it is necessary to have viewers walk closer to the projection surface within the large exhibition space and focus their eyes on the projection surface. In other words, viewers standing far from the projection surface of the real image will see the image as blurry, and even viewers standing near the projection surface will see the image as blurry if their eyes do not focus on the projection surface. Thus, in situations where it is difficult for viewers to intuitively grasp the position of the projected image, such as when a video image is projected in a relatively large exhibition space, there is a concern that the aesthetic effect of the video work and the advertising effect of the signage system will be reduced.

[0007] Therefore, the main objective of the present invention is to extend the visual effects that an aerial display system provides to the observer. Another objective of the present invention is to make it easier for the observer to clearly perceive the real image, even in environments where it is difficult for the observer to intuitively grasp the position where the real image is formed.

[0008] Therefore, the inventors of the present invention diligently considered means to solve the problems of the prior art described above, and as a result, they found that by providing a physical wall with an opening near the image-forming surface of the real image formed by the aerial imaging member, and by projecting image light from a projection device onto that wall, the observer can observe both the aerial image formed near the opening in the wall and the image projected onto the non-opening part of the wall. In addition, they found that since the wall and the image-forming surface are simultaneously within the observer's field of view, the observer can more easily intuitively grasp the image-forming position of the real image. Based on these findings, the inventors realized that the problems of the prior art could be solved, and thus completed the present invention. Specifically, the present invention has the following configuration or process.

[0009] The first aspect of the present invention relates to an aerial display system. The aerial display system comprises a display device, an aerial imaging member, a wall, and a projection device. The display device displays an image. In this specification, "image" includes both moving images and still images, or either one. The aerial imaging member refracts the image light incident from the display device to form an image in the air. As the aerial imaging member, any known type of optical element capable of forming a real image in the air can be used. As the aerial imaging member, in addition to the two-plane orthogonal reflector method described above (Patent Document 1), for example, a retroreflective method combining a half mirror and a retroreflective sheet, a concave mirror method combining a concave mirror and a half mirror, or a lens array method using a microlens array can be used. The wall has an opening through which at least a portion of the first image light from the aerial imaging member passes. This wall is configured to partition the exhibition space between the observer side and the display device side. The shape of the opening is not limited to a rectangle, but can also be circular, elliptical, triangular, pentagonal, or a polygon with more sides. Furthermore, the projection device is positioned on the same side (i.e., the back side of the wall) as the display device and the aerial image forming member, with the wall as the boundary. At least a portion of the wall onto which the second image light is projected is configured to transmit the second image light at least partially.

[0010] As described above, by forming an image of light from the display device in the air near the opening in the wall, and simultaneously projecting an image from the projection device onto parts of the wall other than the opening, it becomes possible to create a visual expression that seamlessly integrates the aerial image and the wall projection. For example, by projecting an image onto the wall so as to be continuous with the aerial image visible through the opening, a visual effect can be created in which part of the image appears to be leaping out of the wall and floating. Also, by projecting related images around the aerial image visible through the opening, a visual effect can be created in which the aerial image appears to be floating up from the wall. Furthermore, by linking the movement of the images between the aerial image and the wall projection, a dynamic effect can be achieved in which part of the image appears to pop out three-dimensionally from the wall. In this way, by providing a projection device in addition to the display device and aerial image forming member, it becomes possible to create a continuous visual expression that eliminates the awareness of the boundary between the aerial image and the wall projection. This greatly expands the range of effects that the aerial display system can provide and allows for more effective use of the wall.

[0011] Furthermore, in the aerial display system according to the present invention, the projection device is positioned on the same side as the display device and the aerial image forming member, separated by a wall. In this case, at least a portion of the wall onto which the image light is projected from the projection device is configured to transmit the image light at least partially. For example, the portion of the wall onto which the image light is projected from the projection device may be transparent, semi-transparent, or mesh-like. By positioning the projection device on the same side as the display device and the aerial image forming member in this way, the entire system can be integrally installed on the back side of the wall. This provides the observer with only a space to enjoy the image, and the devices do not enter the observer's field of vision. In addition, by positioning the projection device on the back side of the wall and configuring a portion of the wall to allow image light to pass through, the image projected from the projection device is displayed softly on the surface of the wall, achieving an integrated image expression that does not feel out of place with the aerial image. Furthermore, by adjusting the transmittance of the wall, the balance between the aerial image and the wall projection image can be optimized, achieving a more natural continuity of image.

[0012] In the aerial display system according to the present invention, it is preferable that the wall has an opening formed within a range of 300 mm in front of and behind the image-forming surface of the aerial imaging member. The range of 300 mm in front of and behind the image-forming surface is set considering the human clear vision distance (the distance at which the human eye can easily focus). The range of 300 mm in front of and behind the image-forming surface is a range of 600 mm in total length, from a position 300 mm behind the image-forming surface to a position 300 mm in front of the image-forming surface, with respect to the optical axis of the image light perpendicular to the image-forming surface. The aerial display system is configured such that at least a portion of the image light emitted from the display device passes through the opening in the wall. By providing a wall with an opening formed within a range of 300 mm in front of and behind the image-forming surface of the aerial imaging member, the observer's eye naturally focuses on the wall or its opening. As the observer focuses on the opening and views the aerial image through that opening, it becomes easier to intuitively grasp the position of the image-forming surface, and as a result, the aerial image can be clearly seen. Furthermore, since the walls function as a guide for dividing the exhibition space, the presence of the walls allows observers to more accurately perceive the depth of the space, which in turn improves the visibility of the aerial images.

[0013] In the aerial display system according to the present invention, it is preferable that the wall has an upper wall portion and a lower wall portion, with an opening formed between the upper wall portion and the lower wall portion. By forming an opening between the upper wall portion and the lower wall portion in this way, the aerial image appears to float to the observer between the upper and lower walls (opening), and the effect of improving the visibility of the aerial image caused by the presence of the wall can be more effectively exerted. In addition, the human field of view is narrower in the vertical direction (approximately 120 degrees) than in the horizontal direction (approximately 200 degrees), and spatial recognition requires more cues in the vertical direction. Therefore, by providing a wall in the vertical direction, the observer can grasp the extent of the space more accurately, and the presence of the aerial image located in that space can be further emphasized.

[0014] In the aerial display system according to the present invention, it is preferable that at least a portion of the periphery of the opening of the wall is formed at an acute angle. By forming the periphery of the opening of the wall at an acute angle in this way, the boundary of the opening can be made less conspicuous. That is, compared to the case where the periphery of the opening is thick and cut at a right angle, the presence of the wall is reduced when the periphery is formed at an acute angle, and the boundary between the aerial image and the surrounding space becomes more natural. In particular, when projecting an image onto the periphery of the opening, the image gradually darkens in the part formed at an acute angle, so it is possible to suppress the appearance of a step-like difference at the boundary between the aerial image and the wall surface. As a result, the visual continuity between the space in which the aerial image is floating and the wall surface is improved, and the effect of integrated image expression can be enhanced.

[0015] The aerial display system according to the present invention preferably further comprises a detection device and a control device. The detection device detects the presence or position of an object within a predetermined range in front of and behind the image-forming surface of the aerial imaging member. Examples of detection devices include infrared sensors, three-dimensional sensors, depth sensors, stereo cameras, ToF cameras, and ultrasonic sensors. The control device controls the image displayed by the display device according to the detection results of the detection device. By providing a detection device and a control device in this way, the image can be changed interactively in response to the observer's actions. For example, if an observer reaches out to the aerial image, the aerial image may change in response to that action, or the aerial image and the wall projection image may change in conjunction. The predetermined range in front of and behind the image-forming surface can be selected from, for example, a range of 500 mm in front of and behind the image-forming surface. This detection range corresponds to the range of motion in which a person would naturally reach out to touch the aerial image, allowing the observer to interact with the aerial image with effortless and intuitive movements. As a result, by detecting the observer's natural movements and controlling the image, a more attractive and interactive visual expression can be realized.

[0016] In the aerial display system according to the present invention, the aerial imaging member is preferably a two-plane orthogonal reflector type. An aerial imaging member of this type includes a first group of reflective surfaces and a second group of reflective surfaces having reflective surfaces in a direction substantially orthogonal to the first group of reflective surfaces. Substantially orthogonal means in the range of 85 to 95 degrees. The aerial imaging member is configured to form an aerial image by sequentially reflecting the image light incident from the display device with the first group of reflective surfaces and the second group of reflective surfaces. By adopting this two-plane orthogonal reflector type, a bright and clear aerial image can be formed with high light utilization efficiency. Furthermore, this type can be realized with a more compact configuration compared to other types (for example, retroreflection type or concave mirror type), thus increasing the flexibility of the overall system installation.

[0017] A second aspect of the present invention relates to an aerial imaging method. In the aerial imaging method according to the present invention, a display device displays an image (display step). An aerial imaging member refracts the first image light incident from the display device to form an image in the air (imaging step). A wall is provided through which at least a portion of the first image light passes, and a projection device projects a second image light onto the portion of this wall other than the opening (projection step). Here, the projection device is located on the same side as the display device and the aerial imaging member, with the wall as the boundary. At least a portion of the wall onto which the second image light is projected is configured to transmit the second image light at least partially.

[0018] The primary objective of this invention is to enhance the visual effects that an aerial display system provides to the observer. Furthermore, this invention makes it easier for the observer to clearly perceive the real image, even in environments where it is difficult for the observer to intuitively grasp the position of the image, such as a relatively large exhibition space.

[0019] Figure 1 is a schematic side view showing the overall configuration of an aerial display system according to one embodiment of the present invention. Figure 1 is a front view of the aerial display system shown in Figure 1, schematically illustrating the projection of images onto a wall and the formation of images in the air. Figure 2 is a block diagram showing the functional configuration of the control system of an aerial display system according to one embodiment of the present invention. Figure 3 shows the division of video content, where (a) shows the entire video content, (b) shows the image output from the projection device, and (c) shows the image output from the display device.

[0020] The following describes embodiments for carrying out the present invention with reference to the drawings. The present invention is not limited to the embodiments described below, but also includes modifications made to the embodiments described below within the scope that would be obvious to those skilled in the art.

[0021] Figure 1 shows the overall configuration of an aerial display system 100 according to one embodiment of the present invention. Figure 2 schematically shows a front view of the aerial display system 100 according to this embodiment. Furthermore, Figure 3 shows the functional configuration mainly of the control system of the aerial display system 100 according to this embodiment. As shown in Figure 1, the aerial display system 100 according to this embodiment mainly comprises an aerial display device 10, a wall 20, a projection device 30, and a detection device 40. In addition, as shown in Figure 3, the aerial display system 100 further comprises a control device 50, which controls the aerial display device 10, the projection device 30, and the detection device 40.

[0022] For example, as shown in Figures 1 and 2, the aerial display system 100 according to this embodiment is intended to be implemented in an exhibition space where the height from the floor to the ceiling exceeds the height of a typical adult (1400 to 2000 mm). In such an exhibition space, the aerial display system 100 projects a real image of the video in mid-air in front of the line of sight of an observer in a standing position, allowing the observer to view the real image formed in mid-air. In addition, this system projects an image onto the wall 20, allowing the observer to view this projected image along with the real image formed in mid-air. The height from the floor to the ceiling is not particularly limited, but is preferably 2000 mm or more, and particularly preferably 2500 to 4000 mm. The width of the exhibition space, particularly the width of the wall 20, is preferably 1000 mm or more, and particularly preferably 1500 to 4500 mm. The aerial display system 100 according to this embodiment can be suitably used, for example, in exhibition facilities such as art museums and showrooms, or in commercial facilities for displaying video works or advertisements. From the observer's perspective, the image projected onto the wall 20 and the aerial image seen through the opening 21 appear as a single unit, thus achieving a three-dimensional representation with a sense of depth in the entire image. Furthermore, observers can enjoy the image while freely moving in front of the wall 20, and multiple observers can observe simultaneously. The appropriate viewing position for observers is preferably within a range of 500 to 3000 mm in front of the wall 20.

[0023] The aerial display device 10 is a device for forming a real image of an image in the air within the exhibition space. The aerial display device 10 mainly comprises a display device 11, an aerial image forming member 12, and a support 13. The display device 11 outputs image light by displaying an image, and the aerial image forming member 12 uses this image light to form a real image in the air. The support 13 holds the aerial image forming member 12 in a predetermined position and orientation. These components are arranged together on the back side of the wall 20, which will be described later.

[0024] The display device 11 can be a general-purpose display such as a liquid crystal display or an organic EL display. In this embodiment, the display device 11 is positioned along the floor and outputs image light upwards (towards the ceiling). The placement of the display device 11 is not limited to this; for example, it can be mounted on the ceiling or leaned against a side wall by bending the optical path of the image light using an optical element such as a mirror. As mentioned above, in this embodiment, it is assumed that an image will be projected onto a relatively large exhibition space, so it is preferable to use a large display for the display device 11. The size of the aerial image depends on the display size of the display device 11, so for example, it is preferable that the diagonal length of the display screen of the display device 11 is 40 inches or more, and particularly preferable that it is 50 to 80 inches. Also, the display device 11 is not limited to a single display, but may be configured by connecting multiple displays together.

[0025] The aerial imaging member 12 is an optical element that refracts the image light incident from the display device 11 to form a real image in the air. As the aerial imaging member 12, any known type of optical element capable of forming a real image in the air can be used, such as a two-plane orthogonal reflector system, a retroreflection system, a concave mirror system, or a lens array system. The two-plane orthogonal reflector system includes a first group of reflective surfaces and a second group of reflective surfaces having reflective surfaces in a direction substantially orthogonal to the first group of reflective surfaces (substantially orthogonal means in the range of 85 to 95 degrees). In this system, a real image is formed in the air by sequentially reflecting the image light incident from the display device 11 by the first group of reflective surfaces and the second group of reflective surfaces. Such a two-plane orthogonal reflector system has the advantage of being able to form a bright and clear real image with high light utilization efficiency. Furthermore, the retroreflective method has a configuration that combines a half-mirror and a retroreflective sheet, and a real image is formed by returning the image light reflected by the half-mirror back in the incident direction by the retroreflective sheet. This method has the advantage of having a simple structure and being easy to manufacture. The concave mirror method has a configuration that combines a concave mirror and a half-mirror, and the image light focused by the concave mirror forms a real image via the half-mirror. This method has the advantage of being able to form a large real image. The lens array method has a configuration in which multiple microlenses are arranged regularly, and a real image is formed by focusing the image light with each microlens. This method has the advantage of being easy to make thin. In particular, the two-plane orthogonal reflector method has the advantage of being able to form a bright and clear real image with high light utilization efficiency, and can be realized with a compact configuration compared to other methods, so it is preferable to adopt this method in this embodiment.

[0026] Furthermore, the aerial imaging member 12 is preferably a thin, plate-like structure, and its thickness can be, for example, about 10 to 50 mm. The planar size of the aerial imaging member 12 is preferably about the same as or greater than the display screen size of the display device 11. Specifically, this is determined based on the display screen size of the display device 11, the distance between the display device 11 and the aerial imaging member 12, and the inclination angle of the aerial imaging member 12. For example, if the display screen of the display device 11 is 50 inches, the diagonal length of the aerial imaging member 12 is preferably 60 inches or more. In this embodiment, the aerial imaging member 12 is installed by a support 13 at an angle of approximately 45 degrees to the floor. This inclination angle is set to image the image light emitted from the display device 11 at an appropriate position and can be adjusted as appropriate within the range of 30 to 50 degrees. As shown in Figure 1, the inclination angle of the aerial imaging member 12 is such that it gradually tilts backward (towards the opposite side of the observer) in the direction rising from the floor.

[0027] Furthermore, as shown in Figure 1, the image light output from the display device 11 diffuses as it enters the aerial imaging member 12. The aerial imaging member 12 then refracts the image light incident from the display device 11 at a predetermined angle, thereby forming a real image in the air. Here, the angle of incidence and the angle of emission of the image light with respect to the aerial imaging member 12 are equal. In the example shown in Figure 1, the aerial imaging member 12 is positioned such that the refraction angle of the image light emitted perpendicularly from the display device 11 is 90 degrees. That is, the image light traveling perpendicularly from such a display device 11 enters the aerial imaging member 12 at an angle of 45 degrees and exits at an angle of 45 degrees equal to the angle of incidence. In this way, all the image light emitted diffusely from the display device 11 is refracted by the aerial imaging member 12 and then converges on the same plane to form a real image. In this specification, the plane on which the imaging points of each image light are aligned is called the "imaging plane," and is shown in Figure 1 by the dotted line labeled P. Furthermore, the distance from the position where the image light enters the aerial imaging member 12 to the display surface of the display device 11 (incident light path length L) ｉ ) and the distance from the position where the image light is emitted from the aerial imaging member 12 to the imaging plane P (the length of the emitted light path L). ｏ ) become equal. Thus the incident optical path length L ｉand the output light path length L ｏ As these two values ​​become equal, a real image the same size as the display surface of the display device 11 is formed on the imaging plane P. The real image thus formed is perceived by the observer as if the display surface of the display device 11 were located at the position of the imaging plane P.

[0028] Furthermore, the aerial imaging member 12 is preferably made of a material with high light transmittance, such as acrylic resin or glass. Acrylic resin is particularly suitable because it has excellent processability and sufficient transmittance (90% or more). In the case of a two-plane orthogonal reflector type aerial imaging member 12, the group of reflective surfaces included in the member can be formed by depositing a thin metal film such as aluminum. The group of reflective surfaces needs to be formed with high precision so that the surfaces are orthogonal to each other (within an error of ±5 degrees).

[0029] The support 13 is a member for holding the aerial imaging member 12 in a predetermined position and orientation. The support 13 may have various adjustment mechanisms to precisely adjust its relative position to the aerial imaging member 12. For example, the support 13 may be equipped with a height adjustment mechanism for adjusting the height of the aerial imaging member 12, a position adjustment mechanism for adjusting the front-to-back position, and an angle adjustment mechanism for adjusting the tilt angle. These adjustment mechanisms allow for fine-tuning of the position and orientation of the aerial imaging member 12 to an optimal state, for example, using alignment markers displayed on the display device 11. The support 13 has a fixing part that can be fixed to the floor or side wall of the exhibition space, and is structured to stably hold the aerial imaging member 12.

[0030] The wall 20 is constructed to separate the exhibition space into the installation space for the aerial display device 10 and the space where observers can enter. In this embodiment, the wall 20 plays a role in making the real image formed by the aerial display device 10 easily visible to observers. In addition, the wall 20 also serves as a screen onto which images are projected by the projection device 30. By combining the aerial image formed by the aerial display device 10 and the wall projection image by the projection device 30 in this way, various video effects can be realized.

[0031] As shown in Figures 1 and 2, in this embodiment, the wall 20 is mainly composed of an upper wall portion 22 and a lower wall portion 23, with an opening 21 formed between them. The image light output by the display device 11 of the aerial display device 10 is refracted by the aerial imaging member 12 and reaches the observer through the opening 21 of the wall 20. At this time, the arrangement of the aerial display device 10 and the wall 20 is adjusted so that the imaging surface P of the aerial imaging member 12 is formed near the opening 21 of the wall 20. Specifically, as shown in Figure 1, the upper wall portion 22 and the lower wall portion 23 are configured to be aligned in a straight line perpendicular to the floor, and it is most preferable that the imaging surface P of the aerial imaging member 12 is formed to coincide with the straight line connecting the upper wall portion 22 and the lower wall portion 23. In other words, it is preferable that the imaging surface P is formed within the opening 21 formed between the upper wall portion 22 and the lower wall portion 23, on the extension of the upper wall portion 22 and the lower wall portion 23. Furthermore, since the upper wall portion 22 and the lower wall portion 23 have thickness, the thickness range of these upper and lower walls may be considered as the thickness of the opening 21, and the image-forming surface P may be designed to be formed within this thickness range of the opening 21. In this way, by forming the image-forming surface P within the opening 21 between the upper wall portion 22 and the lower wall portion 23, the distance to the wall 20 and the distance to the image-forming surface P will be the same as seen from the observer's perspective. Also, both the wall 20 and the image-forming surface P will be in the observer's field of view at the same time. As a result, the observer can more easily intuitively grasp the image point P by relying on the wall 20, and thus more easily see the real image of the image that appears on the image-forming surface P clearly.

[0032] As described above, it is most preferable that the image plane P is formed within the opening 21 of the wall 20 (on the extension of the upper wall portion 22 and the lower wall portion 23). However, since it is difficult to precisely adjust the position of the image plane P, a distance may exist between the image plane P and the opening 21 of the wall 20. Specifically, the opening 21 (the extension of the upper wall portion 22 and the lower wall portion 23) should be located within a range of 300 mm in front of and behind the image plane P, using the optical axis of the image light traveling perpendicular to the image plane P as a reference. In other words, the range of 300 mm in front of and behind the image plane P means a range of 600 mm in total length, from 300 mm behind the image plane to 300 mm in front of the image plane, with respect to the optical axis of the image light perpendicular to the image plane P. Thus, it is permissible for the image plane P to be shifted by 300 mm in front of and behind the opening 21. This numerical range was determined considering the characteristics of human vision. In other words, the human eye has a distance called the clear vision distance, at which eye strain is minimized and the ability to focus on an object is most comfortable. This clear vision distance is generally known to be approximately 250 mm to 300 mm, and within this distance range, the burden of focusing is reduced, and stereoscopic vision due to binocular parallax functions effectively. Furthermore, within this distance range, humans can intuitively grasp space. Therefore, by forming the aperture 21 within a range of 300 mm in front of and behind the image plane P, the observer's eyes, when looking at the wall 20, naturally focus on the aperture 21, and as a result, the real image formed on the image plane P within or near the aperture 21 is also more easily seen clearly by the observer. However, it is preferable that the distance between the image plane P and the aperture 21 be as close as possible, more preferably 0 to 250 mm or 0 to 200 mm, and particularly preferably 0 to 150 mm or 0 to 100 mm.

[0033] Furthermore, the vertical height of the wall opening 21 (the straight-line distance from the lower end of the upper wall portion 22 to the upper end of the lower wall portion 23) can be appropriately set according to the size of the display device 11 (the size of the image-forming surface P). For example, the height of the opening 21 may be set to match the vertical width of the display screen of the display device 11, or it may be set to be slightly smaller than the vertical width of the display screen. Specifically, the height of the wall opening 21 can be set to, for example, about 300 to 1000 mm. Also, the horizontal width (horizontal length) of the opening 21 can be appropriately set according to the size of the display device 11 (the size of the aerial image). For example, the width of the opening 21 may be set to match the horizontal width of the display screen of the display device 11, or it may be slightly smaller than the vertical width of the display screen. In this way, the wall opening 21 is designed to match the size of the display device 11. Preferably, the size of the wall opening 21 and the image-forming surface P are substantially the same, and the opening 21 and the image-forming surface P overlap without excess or deficiency. However, for example, the image plane P may be larger than the aperture 21 and partially overlap the aperture 21, or the image plane P may be smaller than the aperture 21 and there may be a gap formed between the image plane P and the periphery of the aperture 21.

[0034] Furthermore, it is preferable that the periphery of the opening 21 in the wall 20, that is, the lower end of the upper wall portion 22 and the upper end of the lower wall portion 23, be formed at an acute angle θ, as shown in the enlarged view of Figure 1. Specifically, the angle θ of the periphery of the opening 21 is preferably 80 degrees or less, and preferably 60 degrees or less or 45 degrees or less. The lower limit of the angle θ is not particularly limited, but for example, it may be 10 degrees or more or 20 degrees or more. By forming the periphery of the opening 21 at an acute angle in this way, the boundary of the opening 21 can be made less conspicuous. That is, compared to the case where the periphery of the opening 21 is thick and cut at a right angle, the presence of the wall 20 is reduced in the case of a periphery formed at an acute angle, and the boundary between the aerial image and the surrounding space becomes more natural. Also, as will be described later, when projecting an image near the periphery of the opening 21, the image gradually darkens in the part formed at an acute angle, so it is possible to suppress the appearance of a step-like difference at the boundary between the aerial image and the wall surface. This improves the visual continuity between the space in which the aerial image is projected and the wall surface, thereby enhancing the effect of integrated visual expression.

[0035] Furthermore, the wall 20 can be formed from a transparent or translucent material such as an acrylic plate, a glass plate, or a resin plate. A preferred example of a resin plate is a milky white translucent polycarbonate plate. In particular, for the part where the image is projected from the projection device 30, it is necessary to transmit and diffuse the image light appropriately, so it is preferable to apply a half-mirror finish with a transmittance of about 30-70% or a light diffusion finish with fine irregularities. The thickness of the plate material forming the wall 20 can be, for example, about 10-30 mm in order to ensure sufficient strength while enabling the formation of a sharp angle around the periphery of the opening 21. Note that the wall 20 is not limited to a single-layer structure, but may also be a multi-layer structure formed by laminating multiple plate materials. In this case, by giving each layer different optical properties, more effective image expression becomes possible. The wall 20 may also be a mesh screen made of metal or plastic. In the case of a mesh screen, appropriate transmittance and projection characteristics can be obtained by adjusting the mesh pitch and aperture ratio. For example, the mesh pitch can be set to approximately 0.5 to 5.0 mm, and the aperture ratio to approximately 30 to 70%. Using such a mesh screen allows for a more open spatial design, as the projected image on the wall 20 can be viewed while also partially viewing the space beyond the wall 20. Furthermore, mesh screens have the advantage of allowing adjustment of the balance between the visibility of the projected image and the visibility of the aerial image by adjusting the mesh pitch and aperture ratio.

[0036] The projection device 30 projects an image onto a part of the wall 20 other than the opening 21. The projection device 30 can be a known projector such as a DLP (Digital Light Processing), liquid crystal, or laser projector. As shown in Figure 1, in this embodiment, the projection device 30, like the aerial display device 10, is positioned in the space opposite the observer, separated by the wall 20. By positioning the projection device 30 behind the wall 20 in this way, it becomes difficult for the observer to directly see the projection device 30, thus enabling a more immersive visual experience. In this embodiment, the projection device 30 is installed on the ceiling as shown in Figure 1, but it is not limited to this; for example, it can also be installed on the back of the wall 20.

[0037] Furthermore, in this embodiment, the projection device 30 is positioned to project an image onto the back surface of the wall 20. However, as mentioned above, if the wall 20 is transparent, semi-transparent, or mesh-like, the observer can see the image projected onto the wall 20 by passing through the wall 20. Although not shown in the illustration, it is also possible to position the projection device 30 to project an image onto the surface (observer side) of the wall 20. In this case, the observer will see the image that has been projected onto and reflected from the wall 20.

[0038] Furthermore, the projection device 30 must be selected to ensure sufficient brightness depending on the material of the wall 20 and the brightness of the ambient light. For example, it is preferable to use a projector with a brightness of about 5,000 to 20,000 lumens. The resolution of the projection device 30 is preferably Full HD (1,920 x 1,080 pixels) or higher, and more preferably 4K (3,840 x 2,160 pixels) or higher. The projection distance from the projection device 30 to the wall 20 can be appropriately set according to the focal length of the projection lens of the projection device 30, but can be, for example, about 1,000 to 3,000 mm. In this embodiment, one projection device 30 is used to project images onto both the upper wall portion 22 and the lower wall portion 23 of the wall 20. Multiple projection devices 30 with this configuration are arranged horizontally on the ceiling. However, separate projection devices 30 may be provided for the upper wall portion 22 and the lower wall portion 23.

[0039] The detection device 40 is a sensor for detecting the presence or position of an object (mainly the observer's hand or fingers) within a predetermined range of the imaging surface P of the aerial imaging member 12. As the detection device 40, known sensors such as an infrared sensor, a three-dimensional sensor, a depth sensor, a stereo camera, a ToF (Time of Flight) camera, or an ultrasonic sensor can be used. As shown in Figure 1, in this embodiment, the detection device 40 is positioned in the space on the same side as the observer, separated by the wall 20. This detection device 40 is installed on the ceiling on the front side of the wall 20, and by irradiating detection light along the front of the wall 20, it can detect the presence or position of an object approaching the front of the wall 20. By positioning the detection device 40 on the front side of the wall 20 in this way, it is possible to detect an observer approaching the wall 20 at an early stage. However, the detection device 40 may also be positioned on the back side of the wall 20 (the same side as the aerial display device 10). In this case, the detection device 40 will detect the observer's hand or fingers that have entered the back side of the wall 20 through the opening 21. Furthermore, the detection device 40 can be installed at any location, such as the floor or ceiling, as long as it can sufficiently cover the space near the imaging surface P.

[0040] Furthermore, the detection range of the detection device 40 is set to, for example, a range of 500 mm in front of and behind the image plane P. This detection range corresponds to the range of motion in which an observer would naturally reach out and try to touch the aerial image. In other words, it is set to a range that an observer can easily reach when reaching out from a stationary position to interact with the aerial image. Preferably, the detection device 40 has a positional resolution of at least 10 mm within this range, and the detection response time is preferably such that it can detect at a frame rate of, for example, 30 frames / second or more, so that it can track human movements in near real time. By satisfying these performance requirements, a natural response to the observer's intuitive movements can be achieved.

[0041] Note that the detection device 40 may be capable of detecting not only the presence and position of an object in the space near the imaging plane P but also the movement and gestures of the object. For example, it can also detect the approaching speed and moving direction when the observer's hand approaches the opening 21, as well as tapping and swiping operations with the fingertips. In addition, it can also detect the movement of multiple fingers, such as pinch-in and pinch-out operations. Thus, the detection device 40 can detect various operations and gestures of the observer, and these detection results are reflected in the video control of the display device 11 and the projection device 30 via the control device 50. Specifically, when the observer approaches with a hand, the virtual image in the air retreats; when tapping with a finger, the virtual image in the air changes; when swiping, the entire video changes, etc., enabling an interactive video presentation according to the observer's operations.

[0042] As shown in FIG. 3, the control device 50 is connected to the display device 11, the projection device 30, and the detection device 40 of the above-described virtual display device 10 by a wired or wireless LAN or the like, and is responsible for controlling the entire virtual display system 100. For example, the control device 50 generates the video output by the display device 11 and the projection device 30, and executes control to output the video to these respective devices 11 and 30. Further, the control device 50 generates or changes the video to be output to the display device 11 and the projection device 30 based on the detection signal from the detection device 40. The control device 50 may be constituted by a single computer, or it is also possible to distribute the functions executed by this control device 50 to a plurality of computers and construct the control device 50 by these plurality of computers. For example, when the control device 50 is configured in a distributed manner, for example, a video generation server for generating the video for the display device 11 and the projection device 30 and an information processing server for analyzing the input information from the detection device 40 can be provided separately. Note that these respective servers are connected to each other via a network to share the necessary information.

[0043] Furthermore, as shown in Figure 3, the control device 50 has at least a control unit 51 and a storage unit 52. The control unit 51 is composed of, for example, a processor and memory. Examples of processors include a known CPU (Central Processing Unit), GPU (Graphics Processing Unit), and other control circuits. The processor performs predetermined arithmetic processing according to a program stored in memory, and executes various control processes while writing the calculation results to the memory's workspace. The memory is composed of volatile memory such as RAM (Random Access Memory) and is used for the arithmetic processing by the processor described above. The storage unit 52 is mainly an element (storage) for storing video content to be output to the display device 11 and projection device 30, and information used for arithmetic processing in the control unit 51. The storage unit 52 is composed of non-volatile memory such as ROM (Read Only Memory). Video content includes, for example, texture images for aerial projection or wall projection, image sequences for animation, pattern images for generating various visual effects, and parameter information for video effects that change according to the detection results of the detection device 40. Furthermore, the storage unit 52 may store tracking parameters for tracking the position and movement of objects detected by the detection device 40. The storage unit 52 may also store programs used for processing in the control unit 51. For example, a group of programs including image processing programs, video generation programs, and libraries for various calculations may be stored. These programs are executed by the processor of the control unit 51, enabling the control unit 51 to perform various functions.

[0044] Figure 4 shows an example of video content generated by the control unit 51 of the control device 50 and output by the display device 11 and projection device 30, respectively. Figure 4(a) shows an example of the entire video content generated by the control device 50, which is a single video (moving image or still image), and this video can be divided into three parts, A to C. Part A is the part that the projection device 30 projects onto the upper wall portion 22 of the wall 20, Part B is the part that the display device 11 of the aerial display device 10 projects onto an image-forming surface P provided at a position corresponding to the opening 21 of the wall 20 via the aerial image-forming member 12, and Part C is the part that the projection device 30 projects onto the lower wall portion 23 of the wall 20. In this way, in order to express the continuity or unity of the entire video, it is preferable for the control unit 51 to first generate a video with each part linked together, and then divide it into each part. For example, the sense of unity of the entire video can be enhanced by unifying the color tone, adjusting the brightness balance, and synchronizing the timing of movement between each part.

[0045] On the other hand, Figure 4(b) shows an example of an image output from the projection device 30 and projected onto the wall 20. As shown in Figure 4(b), the projection device 30 outputs part A, which is projected onto the upper wall portion 22 of the wall 20, and part C, which is projected onto the lower wall portion 23 of the wall 20, as they are. However, part B, which corresponds to the opening 21 of the wall 20, is not output as is, but is replaced with part B', which represents a blank space (non-emitting portion) or black. This is to prevent double projection near the opening 21 and to ensure the visibility of the aerial image. In other words, in this embodiment, the projection device 30 projects a single image onto the wall 20 by combining part A, part B', and part C. Also, Figure 4(c) shows an example of an image output from the display device 11 and projected onto the wall 20 near the opening 21. As shown in Figure 4(c), the display device 11 extracts and displays only part B from the entire image generated by the control unit 51. Part B is the portion visible to the observer through the opening 21 in the wall 20, and is the most attention-grabbing central element of the entire video. Therefore, it is preferable to place the characteristic parts of the video content in Part B. Characteristic parts include, for example, virtual objects, text, or icon images that the observer would want to touch. Furthermore, since the video in Part B appears to float in the air, adding three-dimensional depth or movement that makes it seem as if it is popping out into the foreground can achieve a more effective visual expression.

[0046] Thus, it is preferable that the control unit 51 divides one video content into a wall projection part (A part, C part) and an aerial imaging part (B part), and outputs them to the projection device 30 and the display device 11 respectively. For example, as shown in FIGS. 1 and 2, in order for the observer to observe the entire wall 20, the observer will visually recognize the video in which parts A to C are connected. However, since there is no physical entity (wall 20) in the B part, which is the most accessible part of the entire video from the observer, the observer can be made to have the illusion that the entire video is floating in the air. Also, when the observer reaches out a hand to the opening 21 of the wall 20, the fingers are detected by the detection device 40, and the video changes. For example, various interactive video expressions are possible, such as the virtual object in the video being deformed in response to the approach of the hand, the virtual object avoiding the hand, the aerial imaging being deformed in response to a tapping motion with a finger, or the entire video scrolling in response to a swipe motion. Therefore, the observer can be made to feel as if they are touching a video without a physical entity and that the video changes due to the contact.

[0047] For example, the video generation process by the control device 50 is executed in the following procedure. First, when the detection device 40 detects the proximity or movement of a physical object (such as the observer's finger), the control device 50 predicts the position of the physical object after a predetermined time based on the detected position, movement direction, and movement speed of the physical object. Next, the relative relationship between the predicted position of the physical object and the virtual object in the currently displayed video is determined. Then, when the approach of the physical object to the virtual object is predicted, an operation pattern for the retreat or avoidance of the virtual object is determined for part or all of the video. At this time, considering the continuity of the video display, the retreat or avoidance operation of the virtual object is executed smoothly over a certain period of time (for example, 0.5 seconds). By repeatedly executing the above process, a natural video change can be realized. Note that as the operation pattern for the retreat or avoidance of the virtual object, an operation in which the video escapes up, down, left, or right, an operation in which the video is deformed, an operation in which the video splits or merges, etc. can be adopted.

[0048] Figures 1 and 2 show an example in which an image is projected from the projection device 30 onto a wall 20 that separates the space where the observer enters from the space where the aerial display device 10 is installed. However, in addition to projecting an image onto this wall 20, it is also possible to project an image from another projection device onto, for example, both or either of the left and right side walls shown in Figure 2.

[0049] In this specification, embodiments of the present invention have been described with reference to the drawings in order to express the content of the present invention. However, the present invention is not limited to the above embodiments, and includes modifications and improvements that are obvious to those skilled in the art based on the matters described in this specification.

[0050] 10...Aerial display device 11...Display device 12...Aerial imaging member 13...Support 20...Wall 21...Opening 22...Upper wall 23...Lower wall 30...Projection device 40...Detection device 50...Control device 51...Control unit 52...Storage unit 100...Aerial display system P...Imaging surface

Claims

1. An aerial display system comprising: a display device for displaying an image; an aerial imaging member for refracting a first image light incident from the display device to form an image in the air; a wall having an opening through which at least a portion of the first image light passes; and a projection device for projecting a second image light onto a portion of the wall other than the opening, wherein the projection device is located on the same side as the display device and the aerial imaging member with respect to the wall, and at least a portion of the portion of the wall onto which the second image light is projected is configured to transmit the second image light at least partially.

2. The display system according to claim 1, wherein the wall has the opening formed within a range of 300 mm in front of and behind the image-forming surface of the aerial image-forming member.

3. The aerial display system according to claim 1, wherein the wall has an upper wall portion and a lower wall portion, and the opening is formed between the upper wall portion and the lower wall portion.

4. The aerial display system according to claim 1, wherein at least a portion of the periphery of the opening of the wall is formed at an acute angle.

5. The aerial display system according to claim 1, further comprising: a detection device for detecting the presence or position of an object within a predetermined range in front of and behind the imaging surface of the aerial imaging member; and a control device for controlling the image displayed by the display device according to the detection result of the detection device.

6. The aerial display system according to claim 1, wherein the aerial imaging member includes a first group of reflective surfaces and a second group of reflective surfaces having reflective surfaces in a direction substantially orthogonal to the first group of reflective surfaces, and is configured to form an aerial image by sequentially reflecting the image light incident from the display device by the first group of reflective surfaces and the second group of reflective surfaces.

7. An aerial imaging method comprising the steps of: a display device displaying an image; an aerial imaging member refracting a first image light incident from the display device to form an image in the air; and a wall provided with an opening through which at least a portion of the first image light passes, and a projection device projecting a second image light onto a portion of the wall other than the opening, wherein the projection device is positioned on the same side as the display device and the aerial imaging member with respect to the wall, and at least a portion of the wall onto which the second image light is projected is configured to transmit the second image light at least partially.

Images