Display control device and display control method
Patent Information
- Application Number
- PCT/JP2026/004187
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-09-16
- Filing Date
- 2026-02-05
- Publication Date
- 2026-09-03
Smart Images
Figure JP2026004187_03092026_PF_FP_ABST
Abstract
Description
Display control apparatus and display control method
[0001] The present disclosure relates to a display control apparatus and a display control method for controlling display of a three-dimensional image.
[0002] For remotely controlling heavy machinery such as excavators, a method of displaying binocular stereoscopic images using "side by side" or the like on a head-mounted display is employed. By wearing a head-mounted display and viewing a three-dimensional image displayed on the head-mounted display, an operator can stereoscopically view a work area operated by the heavy machinery. Therefore, the operator can remotely control the heavy machinery at a location distant from the work area without getting on the heavy machinery. Furthermore, since the operator can perform stereoscopic viewing, the operator can easily perceive a sense of distance in the work area.
[0003] In such a method of remotely controlling heavy machinery using a three-dimensional image, when objects such as humans, animals, vehicles, or bicycles exist in or approach the work area operated by the heavy machinery, it is difficult to immediately recognize the presence or approach of these objects. That is, since the work area is stereoscopically displayed in a central region of the three-dimensional image displayed on the head-mounted display, the operator can stereoscopically recognize the situation of the work area. However, since the left image captured by a left camera and the right image captured by a right camera have different imaging ranges, the left end and right end regions of the three-dimensional image obtained by combining the left image and the right image do not form a correct three-dimensional image.
[0004] For this reason, when an object exists on the left end side or the right end side of the three-dimensional image, or when an object approaches the work area from the left end side or the right end side, an operator wearing a head-mounted display may not immediately notice the presence or approach of the object. Furthermore, in stereoscopic images, distortion occurs in the left end and right end regions, so it is difficult for the operator to accurately recognize the position of an object existing near the work area or approaching the work area.
[0005] Furthermore, if a method is adopted in which sensors for object detection are installed on heavy machinery and a warning is displayed on the stereoscopic image using a GUI (Graphical User Interface) when an object is detected, there is a possibility that the operator may not notice the warning if the display color of the GUI is similar to the color of the object present in the work area.
[0006] Patent Document 1 discloses a method for generating a stereoscopic panoramic image, which involves generating a left panoramic image within a left panoramic image to generate a stereoscopic panoramic image. Patent Document 1 describes that in the left panoramic image, the rear image region captured by the left panoramic camera is replaced with the rear image region captured by the right panoramic camera in the right panoramic image, and this is combined with the left panoramic image to create an image for the left eye.
[0007] Japanese Patent Publication No. 2014-95808
[0008] The technology disclosed in Patent Document 1 aims to provide stereoscopic views of panoramic images and does not address the proper identification of objects that may obstruct the work.
[0009] The purpose of this disclosure is to provide a display control device and a display control method that enable the immediate recognition of objects present in the vicinity of the field of view while a three-dimensional image is being displayed.
[0010] The display control device according to the embodiment includes a control unit that acquires a left image captured of the left region of the field of view and a right image captured of the right region of the field of view, and a stereoscopic display unit that includes a left display unit and a right display unit and displays the image of the field of view in three dimensions. The control unit displays the left image on the left display unit and the right image on the right display unit, determines whether the presence of an object has been detected in the left end region which is to the left of the left region or the right end region which is to the right of the right region, and when it is determined that the object has been detected in the left end region, it generates a left end portion image of the left image and replaces the left region of the right image with the generated left end portion image and displays it on the right display unit, and when it is determined that the object has been detected in the right end region, it generates a right end portion image of the right image and replaces the right region of the left image with the generated right end portion image and displays it on the left display unit.
[0011] The display control method according to the embodiment includes: acquiring a left image captured of the left region of the field of view and a right image captured of the right region of the field of view; displaying the left image on the left display unit of the stereoscopic display unit and the right image on the right display unit of the stereoscopic display unit; determining whether or not an object has been detected in the left end region which is to the left of the left region or the right end region which is to the right of the right region; if it is determined that the object has been detected in the left end region, generating a left end portion image of the left image and replacing the left region of the right image with the generated left end portion image and displaying it on the right display unit; and if it is determined that the object has been detected in the right end region, generating a right end portion image of the right image and replacing the right region of the left image with the generated right end portion image and displaying it on the left display unit.
[0012] With the above configuration, it becomes possible to instantly recognize objects present around the field of view while displaying a three-dimensional image.
[0013] Figure 1 is a block diagram showing the configuration of a display control device according to the first embodiment. Figure 2A is an explanatory diagram showing an operator wearing a head-mounted display, which is the display unit of the display control device. Figure 2B is an explanatory diagram showing the seat in the driver's cab of an excavator, and the right camera and left camera installed on the left and right sides. Figure 3A is an explanatory diagram showing an example of a left image captured by the left camera. Figure 3B is an explanatory diagram showing an example of a right image captured by the right camera. Figure 4A is an explanatory diagram showing a three-dimensional image that can be viewed stereoscopically by an operator wearing a head-mounted display. Figure 4B is an explanatory diagram showing the working range and areas outside the working range of the three-dimensional image shown in Figure 4A. Figure 5A is an explanatory diagram showing the working range and areas outside the working range by changing the left portion of the right image shown in Figure 3B to the leftmost portion of the left image, and extending the left side. Figure 5B is an explanatory diagram showing the working range and areas outside the working range by extending the left side of the three-dimensional image shown in Figure 4A. Figure 5C is an explanatory diagram showing images of the working range and areas outside the working range, with the right portion of the left image shown in Figure 3A replaced with the rightmost portion of the right image. Figure 5D is an explanatory diagram showing images of the working range and areas outside the working range, with the right portion of the three-dimensional image shown in Figure 4A extended. Figure 6 is a flowchart showing the processing procedure of the display control device according to the first embodiment. Figure 7 is a block diagram showing the configuration of the display control device according to the second embodiment. Figure 8 is a flowchart showing the processing procedure of the display control device according to the second embodiment.
[0014] Hereinafter, a display control device according to an embodiment of the present invention will be described with reference to the drawings. Figure 1 is a block diagram showing the configuration of a display control device 100 according to the first embodiment. Figure 2A is an explanatory diagram showing an operator P wearing a head-mounted display 51, which is the display unit 3 of the display control device 100, and operating heavy machinery using an operating device 52, which is the operating unit of a heavy machinery control device (not shown). The operating device 52 is connected to the control unit 2 of the heavy machinery via a communication network (not shown) and transmits operation signals for operating the heavy machinery to the control unit 2 of the heavy machinery. Figure 2B is an explanatory diagram showing the right camera 1R and the left camera 1L installed on the left and right sides of the seat in the driver's seat of an excavator. As shown in Figure 1, the display control device 100 includes a dual-lens camera 1, a control unit 2, and a stereoscopic display unit 3.
[0015] In this embodiment, an example is described in which, when an operator P remotely controls heavy machinery such as an excavator using an operating device 52, the image of the forward field of view (field of view) visible from the operator's cabin of the heavy machinery is displayed in three dimensions on a head-mounted display 51. The image displayed by the display control device 100 according to the first embodiment is not limited to the image of the forward field of view of the heavy machinery, but can also be used for other purposes such as autonomous driving of vehicles and remote control of surgical instruments. The display control device 100 according to the first embodiment is not limited to application to a head-mounted display 51, but can also be applied to other three-dimensional display methods.
[0016] As shown in Figures 1 and 2B, the twin-lens camera 1 comprises a left camera 1L and a right camera 1R. The left camera 1L is installed, for example, on the front left side of the operator's cab of heavy machinery, and captures an image of the left side of the forward field of view. The right camera 1R is installed, for example, on the front right side of the operator's cab of heavy machinery, and captures an image of the right side of the forward field of view. The images captured by the left camera 1L and the right camera 1R are output to the image acquisition unit 21, which will be described later. The left camera 1L captures the left image D1L shown in Figure 3A, which will be described later. The right camera 1R captures the right image D1R shown in Figure 3B, which will be described later. The left camera 1L and the right camera 1R of the twin-lens camera 1 can be configured using cameras equipped with image sensors such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.
[0017] The control unit 2 comprises an image acquisition unit 21, a display control unit 22, a partial image generation unit 23, and a detection unit 24. The image acquisition unit 21, the display control unit 22, the partial image generation unit 23, and the detection unit 24 can be implemented using a microcomputer equipped with a CPU (Central Processing Unit), memory, and an input / output unit. Computer programs for each of the above information processing units (image acquisition unit 21, display control unit 22, partial image generation unit 23, and detection unit 24) are installed on the microcomputer and executed. As a result, the microcomputer functions as each information processing unit. Here, an example of implementing each information processing unit using software is shown, but it may also be configured by preparing dedicated hardware to execute each information processing unit. The information processing executed by each information processing unit may also be configured using separate hardware.
[0018] The video acquisition unit 21 acquires the left image from the image captured by the left camera 1L and the right image from the image captured by the right camera 1R. Figure 3A shows an example of the left image D1L captured by the left camera 1L, and Figure 3B shows an example of the right image D1R captured by the right camera 1R. As shown in Figures 3A and 3B, the left image D1L and the right image D1R each include the image of the central region R0. The left image D1L shown in Figure 3A includes the image of the left region (central region R0, leftmost region R1). The right image D1R shown in Figure 3B includes the image of the right region (central region R0, rightmost region R2). Hereinafter, the image of the leftmost region R1 will be referred to as the "leftmost portion image," and the image of the rightmost region R2 will be referred to as the "rightmost portion image." That is, the left image D1L includes the image of the central region R0 and the leftmost portion image. The right image D1R includes the image of the central region R0 and the rightmost portion image.
[0019] The partial image generation unit 23 extracts the image from a predetermined range on the left side of the left image D1L acquired by the image acquisition unit 21. Here, it is assumed that the leftmost partial image (image of the leftmost region R1) is extracted from the left image D1L shown in Figure 3A. The partial image generation unit 23 also extracts the image from a predetermined range on the right side of the right image D1R acquired by the image acquisition unit 21. Here, it is assumed that the rightmost partial image (image of the rightmost region R2) is extracted from the right image D1R shown in Figure 3B.
[0020] The partial image generation unit 23 generates a modified image to change the left region of the right image D1R when it detects the presence of an object in the leftmost partial image, which is the left side of the left image D1L. Specifically, it generates the leftmost partial image of the leftmost region R1 shown in Figure 3A as a modified image to change the left region of the right image D1R shown in Figure 3B. The partial image generation unit 23 also generates a modified image to change the right region of the left image D1L when it detects the presence of an object in the rightmost partial image, which is the right side of the right image D1R. Specifically, it generates the rightmost partial image of the rightmost region R2 shown in Figure 3B as a modified image to change the right region of the left image D1L shown in Figure 3A.
[0021] The detection unit 24 detects objects in the leftmost region R1 or the rightmost region R2, and objects approaching the central region R0 from the leftmost region R1 or the rightmost region R2, based on the left image D1L and the right image D1R acquired by the image acquisition unit 21. "Objects" refer to obstacles such as pedestrians, animals, vehicles, and bicycles, as well as moving objects such as workers working around heavy machinery, other heavy machinery operated by persons other than operator P, and stationary objects such as materials used in the work. The detection unit 24 may also detect objects that would be an obstacle to work within the range displayed on the stereoscopic display unit 3. The detection unit 24 can detect objects using well-known methods such as millimeter-wave radar, infrared sensors, and LiDAR, which detect objects included in the image. Alternatively, the detection unit 24 may store templates of various objects and detect objects by template matching with the image captured by the dual-lens camera 1.
[0022] The detection unit 24 determines whether an object detected in the leftmost region R1 or the rightmost region R2 is approaching or is predicted to approach the central region R0. Based on this determination, the detection unit 24 detects an object approaching the central region R0 from the leftmost region R1 or the rightmost region R2.
[0023] The detection unit 24 outputs an object detection signal to the partial image generation unit 23 when it detects an object in each region R1 or R2, or when it detects an object approaching the central region R0 from each region R1 or R2. Specifically, when it detects a stationary object or an object approaching the central region R0 in the leftmost region R1 shown in Figure 3A, it outputs a detection signal (hereinafter referred to as the "left object signal") to the partial image generation unit 23. When it detects a stationary object or an object approaching the central region R0 in the rightmost region R2 shown in Figure 3B, it outputs a detection signal (hereinafter referred to as the "right object signal") to the partial image generation unit 23.
[0024] The display control unit 22 outputs the left image D1L and the right image D1R acquired by the image acquisition unit 21 to the stereoscopic display unit 3. When the left object signal is output from the detection unit 24, the display control unit 22 changes the leftmost region of the right image D1R (see Figure 3B) to the leftmost portion image (the image of the leftmost region R1 in Figure 3A). Specifically, as shown in Figure 5A, which will be described later, it generates image D3R by adding the image of region R1 to the leftmost part of the right image D1R. When the right object signal is output from the detection unit 24, the display control unit 22 changes the rightmost region of the left image D1L (see Figure 3A) to the rightmost portion image (the image of the rightmost region R2 in Figure 3B). Specifically, as shown in Figure 5C, which will be described later, it generates image D3L by adding the image of region R2 to the rightmost part of the left image D1L.
[0025] Specifically, the display control unit 22 controls the display of the left image D1L on the left display unit 3L and the right image D1R on the right display unit 3R. If an object exists in the leftmost region R1, or if an object is approaching the central region R0 from the leftmost region R1, the display control unit 22 displays image D3R on the right display unit 3R, which is obtained by replacing the left side of the right image D1R with the leftmost portion of the image. If an object exists in the rightmost region R2, or if an object is approaching the central region R0 from the rightmost region R2, the display control unit 22 displays image D3L on the left display unit 3L, which is obtained by replacing the right side of the left image D1L with the rightmost portion of the image. Details of the process for changing the images will be described later.
[0026] The stereoscopic display unit 3 comprises a left display unit 3L and a right display unit 3R. The left display unit 3L and the right display unit 3R display the left image D1L and the right image D1R, respectively, output from the display control unit 22. The stereoscopic display unit 3 can generate stereoscopic images based on the parallax of the images displayed in each display unit 3L and 3R. That is, as shown in Figure 2A, an operator P wearing the head-mounted display 51 can view the image in the forward field of view visible from the operator's cabin of the heavy machinery in stereo by viewing the image displayed in the left display unit 3L and the image displayed in the right display unit 3R of the stereoscopic display unit 3 with their left and right eyes, respectively. In addition to the aforementioned displays (the left display unit 3L and the right display unit 3R of the stereoscopic display unit 3), the head-mounted display 51 also comprises left and right stereoscopic lenses (not shown), an acceleration sensor and a gyro sensor for tracking the position of the head.
[0027] Next, the details of the process for changing the image in the display control unit 22 will be explained. Figure 4A is an explanatory diagram showing a three-dimensional image D2 that can be viewed in stereo by an operator P wearing a head-mounted display 51. That is, by wearing the head-mounted display 51, operator P can see the three-dimensional image D2 in the forward field of view as shown in Figure 4A, and can operate the heavy machinery with the same sense of distance as if looking forward from the cockpit of the heavy machinery.
[0028] Figure 4B is an explanatory diagram showing the areas within and outside the working range of the three-dimensional image shown in Figure 4A. The three-dimensional image created by combining the left image D1L and the right image D1R shown in Figures 3A and 3B exhibits distortion in the left and right regions, such as changes in aspect ratio or shape, compared to the real image. Specifically, as shown in Figure 4B, a clear stereoscopic image is obtained in region R10, which is within the working range and is the center of the three-dimensional image D2. However, distortion occurs in the three-dimensional image in regions R11 and R12, which are to the left and right of region R10. The distortion is particularly noticeable when a wide-angle lens is used as the lens for the twin-lens camera 1.
[0029] When operator P remotely controls heavy machinery while viewing the 3D image displayed on the head-mounted display 51, the images in the left and right regions R11 and R12 (see Figure 4B), which are outside the working range, are of low importance. In other words, the working range of the heavy machinery is the center of the forward field of view, and operator P remotely controls the heavy machinery while focusing on region R10 within the central working range shown in Figure 4B. Therefore, even if there is some distortion in the left and right regions R11 and R12, which are outside the working range, it does not cause any major problems for operator P's remote control.
[0030] However, if an object is present near the working area R10, or if a person or other work equipment approaches area R10, it is necessary to recognize it immediately. As mentioned above, distortion occurs in the areas R11 and R12, which are outside the working area on the left and right sides of the 3D image D2, making it difficult to immediately recognize an object when it appears in these areas. Furthermore, even if the presence of an object can be recognized, it is difficult to accurately recognize its size and speed of movement. In this embodiment, the presence or approach of an object is addressed as shown below.
[0031] The following describes in detail the processing performed by the display control unit 22 when an object is detected in the leftmost region R1 of the left image (D1L in Figure 3A) or in the rightmost region R2 of the right image (D1R in Figure 3B), or when an object approaching the central region R0 is detected. Figure 5A is an explanatory diagram showing the image D3R (hereinafter referred to as "right-modified image D3R") of the expanded working range and outside the working range, obtained by changing the left portion of the right image D1R shown in Figure 3B to the leftmost portion of the leftmost region R1 of the left image D1L. Figure 5B is an explanatory diagram showing the image D2A of the expanded working range and outside the working range, obtained by extending the left portion of the three-dimensional image D2 shown in Figure 4A. Figure 5C is an explanatory diagram showing the image D3L (hereinafter referred to as "left-modified image D3L") of the expanded working range and outside the working range, obtained by changing the right portion of the left image D1L shown in Figure 3A to the rightmost portion of the rightmost region R2 of the right image D1R. Figure 5D is an explanatory diagram showing image D2B of the area within and outside the working range, which is an extended version of the 3D image D2 shown in Figure 4A, with the right side extended.
[0032] When the detection unit 24 outputs a left object signal, the display control unit 22 performs a process to change the left portion of the right image D1R (image captured by the right camera 1R) to the leftmost portion of the left image D1L (image captured by the left camera 1L). Specifically, it performs a process to change the left portion of the right image D1R shown in Figure 3B to the leftmost portion of the image (image of the leftmost region R1) shown in Figure 3A. As a result, the image D2A shown in Figure 5B is generated from the left image D1L (Figure 3A) and the right modified image D3R (Figure 5A) with the left portion changed. The display control unit 22 does not change the image output to the left display unit 3L, but changes the image output to the right display unit 3R to the right modified image D3R shown in Figure 5A. As a result, the stereoscopic display unit 3 displays the image D2A, which is an expanded version of the left portion of the three-dimensional image D2 shown in Figure 4A, as shown in Figure 5B. As shown in Figure 5B, the image D2A displays the working area (central area R20), the area outside the working area on the left (area R21), and the area outside the working area on the right (area R22). Area R21, although designated as outside the working area, displays a wide area of the image.
[0033] When changing the left portion of the right image D1R to the leftmost portion of the left image D1L (image of region R1), displaying the modified right image D3R on the right display unit 3R prevents the stereoscopic display unit 3 from displaying the left portion of the forward field of view in three dimensions. However, it becomes possible to clearly display objects located in the leftmost region R1, or objects approaching the central region R0 from the leftmost region R1, allowing for immediate recognition of the presence and approach of objects. Furthermore, the size and approach speed of objects can also be easily recognized. When changing the left portion of the right image D1R to the leftmost portion of the left image D1L (image of region R1), the left portion of the right image D1R may be adjusted in advance to match the parallax of the right image D1R using a well-known method for generating viewpoint transformation images using a three-dimensional shape model and texture mapping, thereby providing a natural three-dimensional display of the forward field of view in the stereoscopic display unit 3.
[0034] When the detection unit 24 outputs a right object signal, the display control unit 22 performs a process to change the right portion of the left image D1L (image captured by the left camera 1L) to the rightmost portion of the right image D1R (image captured by the right camera 1R). Specifically, it performs a process to change the right portion of the left image D1L shown in Figure 3A to the right portion of the image (image of the rightmost region R2) shown in Figure 3B. As a result, the image D2B shown in Figure 5D is generated from the right image D1R (Figure 3B) and the modified left image D3L (Figure 5C) with the right portion changed. The display control unit 22 does not change the image output to the right display unit 3R, but changes the image output to the left display unit 3L to the modified left image D3L shown in Figure 5C. As a result, the stereoscopic display unit 3 displays the image D2B, which is an expanded right portion of the three-dimensional image D2 shown in Figure 4A, as shown in Figure 5D. As shown in Figure 5D, the video D2B displays the working area (central area R30), the area outside the working area on the left (area R31), and the area outside the working area on the right (area R32). Area R32, although designated as outside the working area, displays a wide area of the video.
[0035] When changing the right portion of the left image D1L to the rightmost portion of the right image D1R (image of region R2), displaying the modified left image D3L on the left display unit 3L prevents the stereoscopic display unit 3 from displaying the right portion of the forward field of view in three dimensions. However, it becomes possible to clearly display objects in the rightmost region R2, or objects approaching the central region R0 from the rightmost region R2, allowing for immediate recognition of the presence and approach of objects. Furthermore, the size and approach speed of objects can also be easily recognized. When changing the right portion of the left image D1L to the rightmost portion of the right image D1R (image of region R2), the right portion of the left image D1L may be adjusted in advance to match the parallax of the left image D1L using a well-known method for generating viewpoint transformation images using a three-dimensional shape model and texture mapping, thereby ensuring a natural three-dimensional display of the forward field of view in the stereoscopic display unit 3.
[0036] The leftmost region R1 (Figure 3A) and the rightmost region R2 (Figure 3B) can be set to be areas outside the work area when operating heavy machinery. By setting them in this way, it is possible to avoid the work area image being changed after the image has been changed, thus avoiding a significant impact on remote operation using heavy machinery. That is, the partial image generation unit 23 sets the leftmost partial image area and the rightmost partial image area in an area that does not include the central region set in the center of the left-right direction of the forward field of view (field of view). The "field of view" refers to the central region R0, the leftmost region R1, and the rightmost region R2 shown in Figures 3A and 3B above.
[0037] If the heavy machinery is an excavator, for example, the central region R0 shown in Figures 3A and 3B may be set to the range (movable range) in which the excavator's arm rotates left and right. If the heavy machinery does not have an arm, such as a bulldozer, for example, the range of the central region R0 may be set to a range that is twice the width of the vehicle. If applied to a device that operates a vehicle remotely, for example, the range that is twice the width of the vehicle may be set to the central region R0. If applied to a device that operates surgical instruments remotely, for example, the range that is twice the range in which the surgical staff are located may be set to the central region R0.
[0038] When performing the process of changing the left portion of the right image to the leftmost portion of the image, it is necessary to properly align both the left portion of the right image and the leftmost portion of the image. Similarly, when performing the process of changing the right portion of the left image to the rightmost portion of the image, it is necessary to properly align both the right portion of the left image and the rightmost portion of the image. As a method of alignment, the well-known technique of extracting feature points from the left portion of the right image and the feature points from the leftmost portion of the image, and matching these feature points, can be employed.
[0039] Next, the processing procedure of the display control device 100 according to the first embodiment described above will be explained. Figure 6 is a flowchart showing the processing procedure of the display control device 100 according to the first embodiment. First, in step S11 of Figure 6, the video acquisition unit 21 acquires the left image D1L (Figure 3A) and the right image D1R (Figure 3B) of the forward field of view captured by the left camera 1L and the right camera 1R of the dual-lens camera 1.
[0040] In step S12, the detection unit 24 determines whether an object is detected in the leftmost region R1 (Figure 3A) of the left image D1L, or whether an object approaching the central region R0 is detected. If an object is detected (step S12: YES), the process proceeds to step S13; otherwise (step S12: NO), the process proceeds to step S15.
[0041] In step S13, the partial image generation unit 23 generates a modified image to change the left-side region of the right image D1R. Specifically, the partial image generation unit 23 generates the left-side portion image of the left-side region R1 of the left image D1L shown in Figure 3A as a modified image to change the left-side region of the right image D1R shown in Figure 3B. The display control unit 22 performs a process to change the left-side region of the right image D1R to the modified image. As a result, for example, the modified right image D3R shown in Figure 5A is generated.
[0042] In step S14, the display control unit 22 outputs the left image D1L (Figure 3A) to the left display unit 3L. The display control unit 22 also outputs the modified right image D3R, which was modified in step S13, to the right display unit 3R. As a result, the operator P wearing the head-mounted display 51 can clearly see the image of the left side of the forward field of view through the image D2A (Figure 5B) which extends the left side of the working range and the image outside the working range. Therefore, the operator P can immediately recognize when an object is present in the leftmost region R1, or when an object in the leftmost region R1 approaches the central region R0.
[0043] In step S15, the detection unit 24 determines whether an object is detected in the rightmost region R2 (Figure 3B) of the right image D1R, or whether an object approaching the central region R0 is detected. If an object is detected (step S15: YES), the process proceeds to step S16; otherwise (step S15: NO), the process proceeds to step S18.
[0044] In step S16, the partial image generation unit 23 generates a modification image for modifying the right region of the left image D1L. Specifically, the partial image generation unit 23 generates the right-end partial image of the right-end region R2 of the right image D1R illustrated in FIG. 3B as the modification image for modifying the right region of the left image D1L illustrated in FIG. 3A. The display control unit 22 performs processing to change the right region of the left image D1L to the modification image. As a result, for example, the modified left image D3L illustrated in FIG. 5C is generated.
[0045] In step S17, the display control unit 22 outputs the right image D1R (FIG. 3B) to the right display unit 3R. Further, the display control unit 22 outputs the modified left image D3L obtained through the processing of step S16 to the left display unit 3L. As a result, the operator P wearing the head-mounted display 51 can clearly visually recognize the image in the right region of the front visual field area via the image D2B (FIG. 5D) inside and outside the work range with the right portion expanded. Therefore, when an object exists in the right end region R2, or when an object existing in the right end region R2 approaches the central region R0, the operator P can recognize this immediately.
[0046] In step S18, the display control unit 22 outputs the left image D1L to the left display unit 3L and outputs the right image D1R to the right display unit 3R. That is, when the detection unit 24 does not detect the presence of an object, the display control unit 22 displays the left image D1L and the right image D1R on the left display unit 3L and the right display unit 3R, respectively, thereby displaying a three-dimensional image on the head-mounted display 51. The three-dimensional image D2 illustrated in FIG. 4A is displayed on the head-mounted display 51. The operator P wearing the head-mounted display 51 can stereoscopically view the image of the front visual field area (visual field area).
[0047] As described above, when an object is present in the left end region R1 (FIG. 3) in the front visual field, or when an object is approaching the central region R0 from the left end region R1, the display control apparatus 100 according to the present embodiment changes the left region of the right image D1R to a replacement image, and generates a modified right image D3R (FIG. 5A). On the other hand, when an object is present in the right end region R2 in the front visual field, or when an object is approaching the central region R0 from the right end region R2, the display control apparatus changes the right region of the left image D1L to a replacement image, and generates a modified left image D3L (FIG. 5C).
[0048] For this reason, when an object exists around the work area (R10 in FIG. 4B) in the front visual field, or an object approaches the work area, although an operator P cannot stereoscopically view the area outside the work range on the side where the object exists or approaches, the operator P can immediately recognize the presence of the object. Accordingly, risks such as collision with an object can be prevented in advance. Furthermore, when changing the image of the left end portion or the right end portion on the side where the object exists or approaches, by generating the image of the left end portion or the right end portion as a three-dimensional display of the front visual field region in the stereoscopic display unit 3 using an image that matches the parallax of the right image or the left image in advance, by using a well-known method of generating viewpoint-converted images based on a three-dimensional shape model and texture mapping, the presence of an obstacle can be clearly recognized and handled immediately.
[0049] The partial image generation unit 23 sets the left end region R1 and the right end region R2 in a region that does not include the central region R0 (for example, the region R10 serving as the work range shown in FIG. 4B) set at the center in the left-right direction of the front visual field region (visual field range), so that a clear image can be displayed in the work region R10 where remote operation is performed.
[0050] Next, a second embodiment will be described. Figure 7 is a block diagram showing the configuration of the display control device 101 according to the second embodiment. The display control device 101 according to the second embodiment differs from the display control device 100 according to the first embodiment shown in Figure 1 in that it is equipped with a sensor 4 and the detection unit 24 has been changed to a detection unit 24a. The configurations of the sensor 4 and the detection unit 24a will be described below, and other components will be given the same reference numerals as in Figure 1, and their configuration descriptions will be omitted.
[0051] Sensor 4 comprises a left-side sensor 4L and a right-side sensor 4R. The left-side sensor 4L is installed, for example, on the front left side of the operator's cab of heavy machinery. The left-side sensor 4L detects objects in the leftmost region R1 of the forward field of view, and objects approaching the central region R0 from the leftmost region R1. The right-side sensor 4R is installed, for example, on the front right side of the operator's cab of heavy machinery. The right-side sensor 4R detects objects in the rightmost region R2 of the forward field of view, and objects approaching the central region R0 from the rightmost region R2. Ultrasonic sensors, distance measuring sensors, LiDAR, etc., can be used as sensors 4L and 4R. When sensors 4L and 4R detect an object, they output a detection signal to the detection unit 24a.
[0052] When the detection unit 24a outputs an object detection signal from the left sensor 4L and the right sensor 4R, it outputs this detection signal to the partial image generation unit 23.
[0053] Next, the processing procedure of the display control device 101 according to the second embodiment will be described with reference to the flowchart shown in Figure 8. In Figure 8, the processes shown in steps S31, S33, S34, S36, S37, and S38 are the same as the processes shown in steps S11, S13, S14, S16, S17, and S18 in Figure 6. The processes in steps S32 and S35 will be described below.
[0054] In step S32, the detection unit 24a determines whether the left sensor 4L has detected the presence of an object in the leftmost region R1 (Figure 3A) of the left image D1L, or whether an object approaching the central region R0 has been detected. If an object is detected (step S32: YES), the process proceeds to step S33; otherwise (step S32: NO), the process proceeds to step S35. The processes in steps S33 and S34 are the same as the processes in steps S13 and S14 in Figure 3 described above, so their explanation is omitted.
[0055] In step S35, the detection unit 24a determines whether the presence of an object or an object approaching the central region R0 has been detected by the right-side sensor 4R in the right-end region R2 (Figure 3B) of the right image D1R. If an object is detected (step S35: YES), the process proceeds to step S36; otherwise (step S35: NO), the process proceeds to step S38. The processes in steps S36 to S38 are the same as the processes in steps S16 to S18 in Figure 3 described above, so their explanation is omitted.
[0056] Thus, in the display control device 101 according to the second embodiment, similar to the first embodiment described above, if an object is present in the leftmost region R1 of the forward field of view, or if an object is approaching the central region R0 from the leftmost region R1, a right modified image D3R is generated by changing the left region of the right image D1R to a modified image. Similarly, if an object is present in the rightmost region R2 of the forward field of view, or if an object is approaching the central region R0 from the rightmost region R2, a left modified image D3L is generated by changing the right region of the left image D1L to a modified image.
[0057] Therefore, even if an object that is not visible in the working area R10 of the 3D image D2 shown in Figure 4B is present or approaches, the operator P will not be able to see the working area on the side where the object is present or approaching in 3D, but will still be able to detect the presence of the object in advance and prevent risks such as contact with the object. Furthermore, when processing to change the image of the left or right edge portion on the side where the object is present or approaching, the image of the left or right edge portion is pre-configured using a well-known technique of generating viewpoint transformation images using a 3D shape model and texture mapping to create a 3D display of the forward field of view area in the 3D display unit 3 with an image that matches the parallax of the right or left image, making it possible to confirm the presence of the object from the image.
[0058] While embodiments and modifications of the present invention have been described above, it is possible to modify or alter these embodiments and modifications based on the above disclosure. All components of the above embodiments and modifications, and all features described in the claims, may be individually selected and combined, provided that they do not contradict each other.
[0059] The entire contents of Japanese Patent Application No. 2025-027682 (filing date: February 25, 2025) and Japanese Patent Application No. 2025-153202 (filing date: September 16, 2025) are incorporated herein by reference.
[0060] This invention is a technology that can be used when remotely controlling a target device.
Claims
1. A display control device comprising: a control unit that acquires a left image captured of the left region of a field of view and a right image captured of the right region of the field of view; and a stereoscopic display unit that includes a left display unit and a right display unit and displays the image of the field of view in three dimensions, wherein the control unit displays the left image on the left display unit and the right image on the right display unit; determines whether the presence of an object is detected in the left end region which is to the left of the left region or the right end region which is to the right of the right region; when it is determined that the object is detected in the left end region, it generates a left end portion image of the left image, replaces the left region of the right image with the generated left end portion image and displays it on the right display unit; and when it is determined that the object is detected in the right end region, it generates a right end portion image of the right image, replaces the right region of the left image with the generated right end portion image and displays it on the left display unit.
2. The display control device according to claim 1, wherein the control unit determines whether an obstacle approaching the central region of the field of view from the left end region or the right end region has been detected, and when it determines that an obstacle approaching the central region from the left end region has been detected, it replaces the left region of the right image with the left end portion image and displays it on the right display unit, and when it determines that an obstacle approaching the central region from the right end region has been detected, it replaces the right region of the left image with the right end portion image and displays it on the left display unit.
3. The display control device according to claim 1, wherein the control unit detects an object that is an obstacle to work in the area displayed on the three-dimensional display unit.
4. The display control device according to claim 1 or 2, wherein the control unit sets the left end region and the right end region in a region that does not include the central region of the field of view.
5. A display control method comprising: acquiring a left image captured of the left region of the field of view and a right image captured of the right region of the field of view; displaying the left image on the left display unit of the stereoscopic display unit and displaying the right image on the right display unit of the stereoscopic display unit; determining whether or not an object has been detected in the left end region which is to the left of the left region or the right end region which is to the right of the right region; if it is determined that the object has been detected in the left end region, generating a left end portion image of the left image, replacing the left region of the right image with the generated left end portion image and displaying it on the right display unit; and if it is determined that the object has been detected in the right end region, generating a right end portion image of the right image, replacing the right region of the left image with the generated right end portion image and displaying it on the left display unit.