Adjustment method, display system, and computer program

Abstract

Provided are an adjustment method, a display system, and a computer program with which it is possible to more easily adjust an image to be displayed in a predetermined space than in the past. This adjustment method, which is executed by a control device (2) for outputting an image signal to a display device (3) for projecting an image that includes image content onto a projection surface (7) of a predetermined space, comprises: acquiring three-dimensional information that includes a three-dimensional model of the predetermined space, said space having a marker; acquiring a captured image of the predetermined space; acquiring a pattern captured image of the predetermined space onto which a pattern image is projected; generating a conversion relationship in which a first coordinate system of the three-dimensional model and a second coordinate system of the captured image are associated, and a conversion relationship in which the second coordinate system and a third coordinate system of the display device are associated; determining a display area of the image content in the three-dimensional model; and generating an image signal by geometrically correcting the projection content so that the display device projects the image content onto an area of the projection surface that corresponds to the display area.

Description

Adjustment method, display system, and computer program

[0001] This disclosure relates to an adjustment method, a display system, and a computer program for adjusting an image projected by a display device.

[0002] Conventionally, geometric distortion occurs in images projected from a projector onto a screen, and techniques for geometrically correcting images to eliminate this distortion are known. For example, Patent Document 1 discloses an apparatus for reducing geometric distortion of an image projected onto a plane.

[0003] Japanese Patent Publication No. 2016-189560

[0004] The user determines the positions that evenly divide the area for displaying image content and places markers there. Using a control device, the control device can geometrically correct the markers by associating them with a cursor in a predetermined image displayed on that area by the display device, thereby ensuring the linearity of the image content even if the area is not planar. However, accurately determining the positions that evenly divide the area for displaying content in real space is difficult, and placing markers at equal intervals is challenging.

[0005] The purpose of this disclosure is to provide an adjustment method, a display system, and a computer program that can easily adjust images displayed in a predetermined space compared to conventional methods.

[0006] An adjustment method according to one aspect of the present disclosure is an adjustment method performed by a control device that outputs an image signal to a display device that projects an image including image content onto a projection surface in a predetermined space, and includes: acquiring three-dimensional information including a three-dimensional model of a predetermined space having a sign; acquiring an image captured from an imaging device of the predetermined space having a sign; acquiring a pattern image captured from an imaging device of the predetermined space onto which a pattern image has been projected by the display device; generating a first coordinate transformation relationship that associates a first coordinate system of the three-dimensional model with a second coordinate system of the image captured based on the sign; generating a second coordinate transformation relationship that associates a second coordinate system of the pattern image captured with a third coordinate system for projected content projected by the display device based on predetermined features included in the pattern image; determining the display area of ​​the image content in the three-dimensional model based on the coordinates in the first coordinate system; and generating an image signal by geometrically correcting the projected content based on the first and second coordinate transformation relationships so that the display device projects the image content onto the area of ​​the projection surface corresponding to the display area.

[0007] According to this disclosure, it is possible to provide an adjustment method, a display system, and a computer program that can easily adjust images displayed in a predetermined space compared to conventional methods.

[0008] Schematic diagram of a display system according to Embodiment 1 of this disclosure. Schematic diagram of a control device according to Embodiment 1 of this disclosure. Schematic diagram of a display device according to Embodiment 1 of this disclosure. Flowchart showing an example of processing by the control device according to this disclosure. An example of an image of a predetermined space captured by an imaging device. An example of a three-dimensional model of a predetermined space acquired by a three-dimensional shape measuring device. An example of a pattern image captured by an imaging device. An example of a display area determined in a three-dimensional model. An example of a display area determined in a three-dimensional model. An example of an image that the display device intends to project. Another example of a display area determined in a three-dimensional model. Another example of an image that the display device intends to project. Schematic diagram of a display system according to Embodiment 2 of this disclosure. An example of a first pattern image captured by an imaging device. An example of a second pattern image captured by an imaging device. An example of an image that the control device intends to project by a plurality of display devices. A first image and a mask image for the first image projected by the first display device. A second image and a mask image for the second image projected by the second display device.

[0009] The embodiments of this disclosure will be described below with reference to the drawings. However, the configurations described below are merely examples of this disclosure, and this disclosure is not limited to the embodiments described below. The technology in this disclosure is not limited thereto, and various modifications, substitutions, additions, and omissions are possible in other embodiments as long as they do not depart from the technical idea of ​​this disclosure, depending on the design, etc.

[0010] While this disclosure is adequately described in relation to preferred embodiments with reference to the accompanying drawings, various modifications and alterations will be obvious to those skilled in the art. Such modifications and alterations should be understood to be included within the scope of this disclosure as defined by the attached claims.

[0011] In this disclosure, when describing further embodiments, the differences from Embodiment 1 will be described primarily. In this case, components that are the same as or equivalent to those in Embodiment 1 will be denoted by the same reference numerals in the further embodiments. Also, in the further embodiments, descriptions that overlap with those in Embodiment 1 may be omitted.

[0012] In this specification, "projected content" means the content that the control device described later intends to project using the display device described later. "Image content" means the content that the control device generates on the screen using the display device. "Display area" means the area on the three-dimensional model described later that corresponds to the area on the screen where the image content is displayed.

[0013] Conventionally, when a control device displays an image in a predetermined area of ​​a predetermined projection surface via a display device, the control device geometrically corrects the image output to the display device. In this case, by having the user place geometric correction cursors displayed on a virtual projection surface on the control device's UI at equal intervals, the control device can display an image that is linear and undistorted in the predetermined area, regardless of the shape of the projection surface.

[0014] However, if the projection surface is curved, for example, even if the user places cursors at equal intervals on a screen that displays a virtual projection surface in the UI, the distances between the points on the actual projection surface corresponding to the cursors on that screen are not equal. Therefore, the user cannot intuitively place cursors at equal intervals on the UI screen.

[0015] Furthermore, by placing multiple markers at equal intervals on the actual projection surface and associating these markers with the cursor on the UI, the control device can determine the cursor's position. However, it is difficult to place multiple markers at equal intervals on the actual projection surface without errors in distance and angle. If the markers are not placed at equal intervals, associating the cursor with these markers on the UI will result in a distorted image being displayed on the actual projection surface.

[0016] Furthermore, in systems using multiple display devices, each of the multiple display devices projects an image that overlaps with at least a portion of the other display devices, thereby displaying a single image content on the projection surface. In such systems as well, if the cursors cannot be placed at equal intervals, the accuracy of the correction will be low, and it will not be possible to smoothly connect the areas where multiple images overlap.

[0017] Placing multiple markers at equal intervals on the actual projection surface was extremely time-consuming, both to prevent distortion in the projected image and to ensure that multiple images were smoothly connected.

[0018] The adjustment method relating to this disclosure is performed by a control device that outputs an image signal to a display device that projects an image containing image content onto a projection surface in a predetermined space. The adjustment method includes acquiring three-dimensional information including a three-dimensional model of the predetermined space, an image captured of the predetermined space, and a pattern image captured of the predetermined space with a pattern image displayed on it. Markers are provided in the space. The adjustment method includes generating a coordinate transformation relationship between the coordinate system of the three-dimensional model and the coordinate system of the image captured based on the markers. The adjustment method also includes generating a coordinate transformation relationship between the coordinate system of the image captured and the coordinate system of the display device based on feature points included in the pattern image. The adjustment method includes determining a display area for displaying the image content in the three-dimensional model based on the coordinates in the coordinate system of the three-dimensional model. The adjustment method generates an image signal by geometrically correcting the projected content based on the generated coordinate transformation relationship so that the display device displays the image content in the area on the projection surface corresponding to the display area.

[0019] According to this method, the control device can generate a transformation relationship that associates the coordinate system of the 3D model with the coordinate system of the captured image based on the markers. Therefore, without the user having to place markers at equal intervals on the projection surface, the control device can accurately geometrically correct the projected content and display the image content within a predetermined area through the display device. Consequently, the user can easily adjust the projected image through the control device to ensure the linearity of the image content displayed in the predetermined area.

[0020] (Embodiment 1) 1. Configuration [Display System] A display system 1 according to Embodiment 1 of the present disclosure will be described with reference to Figures 1 to 3. Figure 1 is a schematic diagram of a display system 1 according to Embodiment 1 of the present disclosure. The display system 1 comprises a control device 2, a display device 3, an imaging device 4, and a three-dimensional shape measuring device 5. The control device 2 is an information processing device such as a computer or a media server device. The display device 3 is a projector, for example. The control device 2 and the display device 3 can be connected via a cable 6. The display device 3 displays a projected image by projecting an image signal received from the control device 2 onto a screen 7. The screen 7 is an example of a projection surface.

[0021] The control device 2 can transmit control signals to the display device 3 via the cable 6. These control signals include signals for controlling the display device 3. The control device 2 may also transmit audio signals to the display device 3 via the cable 6. The display device 3 can also transmit control signals to the control device 2 via the cable 6. These control signals may include the resolution and frame rate that the display device 3 can display. The control signals may also include the gamma characteristics of the image for the image signal transmitted by the control device 2. For example, the cable 6 conforms to standards such as HDMI (High-Definition Multimedia Interface) (registered trademark) or SDI (Serial Digital Interface).

[0022] [Control device] Figure 2 is a schematic diagram of a control device 2 according to Embodiment 1 of the present disclosure. The control device 2 includes an arithmetic circuit 21, a storage device 22, a communication circuit 23, an input device 24, and an output device 25.

[0023] The arithmetic circuit 21 controls the overall operation of the control device 2. The arithmetic circuit 21 may be configured in a manner in which hardware resources and software cooperate to realize a predetermined function, or in a manner in which a specially designed hardware circuit is used to realize a predetermined function.

[0024] As an example of the former, the arithmetic circuit 21 includes a general-purpose processor such as a CPU or MPU that realizes predetermined processing or functions by executing a program. The arithmetic circuit 21 is configured to communicate with the storage device 22. The arithmetic circuit 21 realizes various functions in the control device 2 by reading and executing arithmetic programs etc. stored in the storage device 22. As an example of the latter, the arithmetic circuit 21 includes FPGAs and ASICs. As can be understood from the above, the arithmetic circuit 21 can be realized using semiconductor integrated circuits such as CPUs, MPUs, GPUs, FPGAs, DSPs, and ASICs.

[0025] The storage device 22 is a storage medium capable of storing various types of information. This information includes programs and data. For example, the storage device 22 stores arithmetic programs for realizing various functions according to this embodiment. The storage device 22 is implemented, for example, by a volatile or non-volatile semiconductor memory such as DRAM, SRAM, flash memory, SSD, HDD, or other storage device, or by an appropriate combination thereof. The data includes, for example, content such as images.

[0026] The communication circuit 23 is an interface device for connecting to other devices or systems via a communication line or communication cable, either wired or wirelessly. The interface device can perform communication compliant with wired communication standards such as USB®, HDMI®, or Ethernet®. The interface device can also perform communication compliant with wireless communication standards such as Wi-Fi®, Bluetooth®, or mobile phone lines.

[0027] The arithmetic circuit 21 can communicate image signals, audio signals, and control signals with external devices through the communication circuit 23. Data communication with external devices such as the display device 3 conforms to standards such as HDMI® and SDI, as described above. The arithmetic circuit 21 can transmit image signals, audio signals, and control signals to the display device 3 via the communication circuit 23. The arithmetic circuit 21 can also receive images captured by the imaging device 4 from the imaging device 4 via the communication circuit 23. Furthermore, the arithmetic circuit 21 can receive three-dimensional information of a three-dimensional model measured by the three-dimensional shape measuring device 5 via the communication circuit 23.

[0028] The input device 24 has the function of inputting information from the user to the control device 2. The input device 24 includes one or more human-machine interface devices. The human-machine interface devices include, for example, a keyboard, a pointing device (mouse, trackball, etc.), a touchpad, etc.

[0029] The output device 25 has the function of outputting information from the control device 2 to the user. The output device 25 includes one or more human-machine interface devices. The human-machine interface devices include, for example, output devices such as a display and a speaker. The human-machine interface devices may also include input / output devices such as a display equipped with an in-cell type touch panel (e.g., a liquid crystal panel or an organic EL panel).

[0030] [Display Device] Figure 3 is a schematic diagram of a display device 3 according to Embodiment 1 of the present disclosure. The display device 3 comprises an arithmetic circuit 31, a storage device 32, a communication circuit 33, an input device 34, an output device 35, and an optical mechanism 36. The arithmetic circuit 31, storage device 32, communication circuit 33, input device 34, and output device 35 can be implemented by the same specific means as the arithmetic circuit 21, storage device 22, communication circuit 23, input device 24, and output device 25 described above using Figure 2. The display device 3 can display image content on the screen 7 by projecting an image including image content onto the screen 7 based on an image signal received from the control device 2.

[0031] The optical mechanism 36 is a mechanism for displaying an image. The optical mechanism 36 comprises a light source such as a laser diode, LED, or lamp, and an optical unit. The optical unit may include a plurality of optical elements. The optical unit includes, for example, a light modulation element such as a liquid crystal element or a DMD (Digital Mirror Device) that modulates the light emitted from the light source, and a projection lens system that guides the image light modulated by the light modulation element to the projection surface.

[0032] [Imaging Device] The imaging device 4 is a camera equipped with an image sensor such as a CCD or CMOS. The imaging device 4 can image a predetermined space including the screen 7. The imaging device 4 can transmit the image acquired by imaging a predetermined section to the control device 2 via the communication circuit 23. The imaging device 4 can take an image in response to input from the user. The imaging device 4 may also be configured to take an image in response to instructions from the arithmetic circuit 21.

[0033] [3D Shape Measurement Device] The 3D shape measurement device 5 is a sensor capable of acquiring a 3D model of an object. The 3D shape measurement device 5 is, for example, a LiDAR (Light Detection And Ranging) capable of measuring the distance to an object by irradiating the object with light of a predetermined wavelength and receiving the reflected light with the sensor. In this embodiment, the 3D shape measurement device 5 irradiates light into a predetermined space including the screen 7 and receives the reflected light. Based on the acquired reflected light data, the 3D shape measurement device 5 generates 3D information including a 3D model of the predetermined space. The 3D information includes a 3D model of the screen 7 in the predetermined space and 3D models of the other parts of the predetermined space. The 3D information is, for example, point cloud data generated based on the acquired reflected light data. The 3D shape measurement device 5 can transmit the generated 3D information to the control device 2 via the communication circuit 23. The 3D shape measurement device 5 can receive input from a user and generate 3D information. The 3D shape measurement device 5 may be configured to receive instructions from the calculation circuit 21 and generate 3D information.

[0034] The three-dimensional shape measurement device 5 may be a device in which a LiDAR and a camera are combined. In such a case, since the three-dimensional shape measurement device 5 can acquire data in which the texture imaged by the camera is combined with the point cloud data, the control device 2 can execute the processes described below using the data.

[0035] In the present embodiment, a label is attached to the screen 7. The label is configured to be distinguishable on the screen 7 in the captured image captured by the imaging device 4. For example, the label has a color different from that of the screen 7. Therefore, a user who checks the captured image with the output device 25 of the control device 2 can identify the label on the screen 7 based on the color. Similarly, the arithmetic circuit 21 can identify the label on the screen 7 based on the color. Further, the label is configured to be distinguishable on the screen 7 in the three-dimensional information generated by the three-dimensional shape measurement device 5. For example, the label has a reflectance different from that of the screen 7. In the three-dimensional information generated by the three-dimensional shape measurement device 5, the data corresponding to the label is different from, for example, the data corresponding to the portions other than the label on the screen 7 in terms of the light intensity. Therefore, a user who checks the three-dimensional information with the output device 25 of the control device 2 can identify the label on the screen 7 based on the data included in the three-dimensional information. Similarly, the arithmetic circuit 21 can identify the label on the screen 7 based on the data included in the three-dimensional information.

[0036] 2. Operations Hereinafter, an outline of the processing by the control device 2 of the display system 1 according to the present embodiment will be described. FIG. 4 is a flowchart showing an example of the processing by the control device 2 according to the present disclosure.

[0037] The arithmetic circuit 21 of the control device 2 acquires three-dimensional information including a three-dimensional model of a predetermined space including the screen 7 having a label from the three-dimensional shape measurement device 5 (S10). Further, the arithmetic circuit 21 acquires a captured image of the predetermined space captured by the imaging device 4 (S11). Further, the arithmetic circuit 21 acquires a pattern captured image of a predetermined space including the screen 7 on which the pattern image is projected by the display device 3 from the imaging device 4 (S12).

[0038] The arithmetic circuit 21 determines a set of coordinates indicating the same points between the 3D model and the captured image based on the markers (S13). Then, the arithmetic circuit 21 generates a transformation matrix for associating the coordinate system of the 3D model with the coordinate system of the captured image based on the coordinates of the points indicating the same points (S14). Thereby, the arithmetic circuit 21 can associate the 3D model with the captured image. The coordinate system of the 3D model is an example of the first coordinate system. The coordinate system of the captured image is an example of the second coordinate system.

[0039] Also, the arithmetic circuit 21 generates a conversion table for associating the coordinate system of the captured image with the coordinate system of the display device 3 based on the feature points included in the pattern image (S15). Thereby, the arithmetic circuit 21 can associate the captured image with the projection content projected by the display device 3. The coordinate system of the display device 3 is an example of the third coordinate system.

[0040] Then, the arithmetic circuit 21 determines a display area for displaying the image content in the 3D model (S16). The arithmetic circuit 21 geometrically corrects the projection content based on the generated transformation matrix and conversion table (S17). Thereby, the arithmetic circuit 21 can geometrically correct the projection content so that the image content is displayed within the area of the screen 7 corresponding to the display area. When the arithmetic circuit 21 geometrically corrects the projection content, it generates an image signal including the geometrically corrected projection content (S18). Then, the arithmetic circuit 21 transmits the image signal to the display device 3.

[0041] When the display device 3 receives the image signal, it projects the projection content onto the screen 7 based on the image signal. Since the projection content is geometrically corrected, the display device 3 can project the projection content onto the area of the screen 7 corresponding to the display area, and the image content is displayed within that area. In Embodiment 1, the projection content corresponds to the same content as the image content.

[0042] Next, each step of the flowchart shown in Figure 4, which is executed by the arithmetic circuit 21, will be described in detail. As described above, the arithmetic circuit 21 acquires a three-dimensional model and an image of a predetermined space having markers (S10, S11). Figure 5 is an example of an image of a predetermined space captured by the imaging device 4. The image in Figure 5 shows a predetermined space including a predetermined surface 8. Surface 8 has a plurality of markers. In this embodiment, surface 8 is provided with four markers a1 to a4 on its outside. In addition, surface 8 is provided with three markers b1 to b3 inside the four markers a1 to a4.

[0043] In this embodiment, the arithmetic circuit 21 receives input from the user via the input device 24 and identifies the screen 7 based on the four markers a1 to a4. The user can identify the four markers a1 to a4 by checking the captured image displayed on the output device 25. The arithmetic circuit 21 can determine the area of ​​the screen 7, for example, by using the four markers a1 to a4 as the four corners of the screen 7. The arithmetic circuit 21 may be configured to detect the four markers a1 to a4 by image analysis and determine the area of ​​the screen 7 based on the detected markers a1 to a4. The arithmetic circuit 21 may be configured to determine the area of ​​the screen 7 as a range to which an arbitrary offset is applied to each marker. In this specification, the provision of markers at the four corners of the screen 7 includes the provision of markers at positions offset in any direction from the corners of the screen 7.

[0044] In this embodiment, the screen 7 shows a projection area 40 on which an image is projected from the display device 3. In this embodiment, the projection area 40 represents the area that the display device 3 can project. In other words, the display device 3 can display any content in any part of the projection area.

[0045] Figure 6 is an example of a three-dimensional model of a predetermined space acquired by the three-dimensional shape measuring device 5. The captured image shown in Figure 5 and the three-dimensional model shown in Figure 6 are images and shapes of the same predetermined space. Markers a1 to a4 shown in Figure 6 correspond to marks a1 to a4 shown in Figure 5, respectively. As described above, marks a1 to a4 have a different reflectivity than the screen 7. Therefore, in the three-dimensional information including data about the three-dimensional model, the data corresponding to marks a1 to a4 is different from the data corresponding to the screen 7. Thus, the user can identify marks a1 to a4 in the three-dimensional model by checking the three-dimensional information displayed on the output device 25.

[0046] Furthermore, the arithmetic circuit 21 acquires a pattern image by capturing a predetermined space on which the pattern image is displayed (S12). The arithmetic circuit 21 transmits an image signal including the pattern image to the display device 3, causing the display device 3 to project the pattern image. The display device 3 projects the pattern image as projection content onto the projection area. The imaging device 4 captures a predetermined space including the pattern image displayed on the screen 7 and acquires it as a pattern image. The arithmetic circuit 21 acquires the pattern image from the imaging device 4.

[0047] Figure 7 is an example of a pattern image 41 captured by the imaging device 4. The pattern image 42 included in the pattern image 41 shown in Figure 7 shows a color pattern combining multiple colors. The pattern image 42 has multiple feature points where four of the multiple colors are adjacent to each other. In the pattern image 42, each of the multiple feature points is configured such that the positions and combinations of the four colors are different. Therefore, each feature point can be uniquely identified based on the positions and combinations of the four colors.

[0048] When the arithmetic circuit 21 acquires a 3D model and an captured image, it determines a set of coordinates that represent the same point between the 3D model and the captured image (S13). In this embodiment, the arithmetic circuit 21 receives input from the user via the input device 24 and determines a plurality of sets of coordinates.

[0049] For example, the arithmetic circuit 21 displays the captured image on the output device 25 and receives an input from the user regarding the position of the marker a1 in the captured image. Next, the arithmetic circuit 21 displays the three-dimensional model via the output device 25 and receives an input from the user regarding the position of the marker a1 in the three-dimensional model. By performing such processing, the arithmetic circuit 21 can determine a pair of the coordinates of the marker a1 in the coordinate system of the three-dimensional model and the coordinates of the marker a1 in the coordinate system of the captured image. The arithmetic circuit 21 similarly processes the markers a2 to a4 as in the case of the marker a1, thereby obtaining a pair of coordinates for each of the markers a1 to a4 in the coordinate system of the three-dimensional model and the coordinate system of the captured image.

[0050] When the arithmetic circuit 21 obtains a set of four points of coordinates for the coordinate system of the three-dimensional model and the coordinate system of the captured image, the arithmetic circuit 21 generates a transformation matrix (S14) for associating the coordinate system of the three-dimensional model with the coordinate system of the captured image using the set of coordinates. In the present specification, the transformation matrix is an example of a first coordinate transformation relationship. For example, the transformation matrix can be defined as in Expression (1).

[0051]

[0052] In Expression (1), x ｘ , ００ , ２２ , ｘ , ｙ , ｘ , Ｍ , ｙ , Ｍ , Ｍ , ｙ , ｚ [[ID=​​​​​​​​​​​​​​​​​​​​​​​​​​

[0053] The calculation circuit 21 calculates T in equation (2) based on equation (1), the coordinates of four sets of captured images representing the same point, and the coordinates of the 3D model. The parameter T does not change unless the position and orientation of the imaging device 4 and the 3D shape measuring device 5 change. Therefore, by calculating T, the calculation circuit 21 can calculate a transformation matrix that allows conversion between the coordinates of the captured images and the coordinates of the 3D model. In the above equation, each coordinate is expressed in a Cartesian coordinate system, but the coordinates used in the transformation matrix are not limited to a Cartesian coordinate system. For example, the calculation circuit 21 may calculate the transformation matrix using a polar coordinate system.

[0054] Furthermore, when the arithmetic circuit 21 acquires a pattern image, it detects the coordinates of feature points in the pattern image. As described above, each feature point in the pattern image can be uniquely identified. The arithmetic circuit 21 obtains a pair of coordinates in the coordinate system of the display device 3 and coordinates in the coordinate system of the captured image by associating the coordinates of feature points in the pattern image with the coordinates of corresponding feature points in the pattern image. When the arithmetic circuit 21 obtains a plurality of pairs of coordinates for the coordinate system of the display device 3 and the coordinate system of the captured image, it uses these pairs of coordinates to generate a conversion table that associates the coordinate system of the display device 3 with the coordinate system of the captured image (S15). In this specification, the conversion table is an example of a second coordinate transformation relationship.

[0055] The arithmetic circuit 21 can generate a more accurate conversion table the more sets of feature points it uses. The arithmetic circuit 21 can generate a highly accurate conversion table by using a predetermined percentage or more of the feature points included in the pattern image. For example, the arithmetic circuit 21 can generate a highly accurate conversion table by using 40% or more of the feature points included in the pattern image.

[0056] In this embodiment, the arithmetic circuit 21 displays a color pattern as a pattern image on the display device 3, but the pattern image is not limited to a color pattern, and any image containing any pattern can be used. For example, the pattern image may be a Gray code pattern or a phase shift pattern. The predetermined ratio described above may change depending on the pattern image.

[0057] In this embodiment, the correspondence between the coordinate system of the display device 3 and the coordinate system of the captured image is determined by a conversion table. By using the conversion table for correspondence, the calculation circuit 21 can establish a correspondence between the coordinate system of the display device 3 and the coordinate system of the captured image not only when the screen 7 is flat, but also when the screen 7 is curved. The calculation circuit 21 may be configured to improve the interpolation accuracy between feature points by accepting input of the screen shape from the user.

[0058] The arithmetic circuit 21 determines a display area in the three-dimensional model that corresponds to the area on the screen 7 where the image content is displayed (S16). For example, the arithmetic circuit 21 can accept a specification of a display area from the user and determine the display area. In this embodiment, the user inputs the markers b1 to b3 shown in Figure 5 to the control device 2 via the input device 24. The arithmetic circuit 21 determines the display area based on the coordinates of the three input points of markers b1 to b3. The arithmetic circuit 21 may be configured to detect the three markers b1 to b3 by image analysis and determine the display area based on the detected markers b1 to b3. The arithmetic circuit 21 may be configured to determine the display area as a range with an arbitrary offset applied to each marker. In this specification, the determination of the four corners of the display area based on three specified points includes determining the corners of the display area at positions offset in an arbitrary direction from each of the three specified points.

[0059] Figures 8A and 8B show examples of display areas 43 determined in a three-dimensional model. Figures 8A and 8B show a three-dimensional model 7A corresponding to the three-dimensional shape of the screen 7. The display area 43 is defined such that three markers b1 to b3 are part of the four corners of the display area 43. The display area 43 may also be defined as an area based on a range with an arbitrary offset applied to the markers. As described above, in this embodiment, the three-dimensional model 7A is point cloud data. As shown in Figure 8A, the display area 43 is defined on the three-dimensional model 7A. The calculation circuit 21 may define a part of the point cloud data as the display area 43. Alternatively, the calculation circuit 21 may add surface data to the point cloud data and define a predetermined area of ​​that surface as the display area 43.

[0060] Once the display area 43 is determined, the arithmetic circuit 21 geometrically corrects the projected content projected by the display device 3 based on the generated transformation matrix and transformation table (S17). First, the arithmetic circuit 21 determines the display area division coordinates that equally divide the display area 43 in the coordinate system of the three-dimensional model. In this specification, the display area division coordinates are an example of first division coordinates.

[0061] For example, if the shape of the screen 7 is planar, the calculation circuit 21 calculates coordinates that equally divide the straight line between markers b2 and b3. The calculation circuit 21 also calculates coordinates that equally divide the straight line between markers b1 and b2. The calculation circuit 21 determines these coordinates as display area division coordinates that equally divide the display area 43 in the coordinate system of the 3D model. From these coordinates, the calculation circuit 21 can generate lines along the direction between markers b1 and b2 or between markers b2 and b3, and generate a mesh that divides the display area. The calculation circuit 21 may also determine each intersection of the mesh as a display area division coordinate.

[0062] Furthermore, as shown in Figures 8A and 8B, the calculation circuit 21 may set predetermined planes 44A and 44B that intersect with the display area 43, and use these predetermined planes 44A and 44B to determine the display area division coordinates. For example, the calculation circuit 21 can calculate coordinates that equally divide the intersection lines 45A and 45B between the predetermined planes 44A and 44B and the display area 43, and determine these coordinates as the display area division coordinates.

[0063] The predetermined plane 44A is, for example, a horizontal plane. The predetermined plane 44B is, for example, a vertical plane. The predetermined plane 44A may be a plane parallel to the floor surface of the predetermined space. The predetermined plane 44B may be a plane perpendicular to the floor surface. The calculation circuit 21 can determine the floor surface from the three-dimensional model included in the three-dimensional information. The predetermined plane 44A may be a plane passing through markers b2 and b3. The predetermined plane 44B may be a plane passing through markers b1 and b2. Calculating coordinates to equally divide the intersection lines 45A and 45B may also mean calculating coordinates to divide the intersection lines 45A and 45B along the shape of the three-dimensional model 7A into equal lengths. The calculation circuit 21 can extract the intersection lines between the predetermined planes 44A and 44B placed at each coordinate and the display area 43, and generate a mesh using the extracted multiple intersection lines.

[0064] The arithmetic circuit 21 calculates the image division coordinates in the coordinate system of the captured image corresponding to the display area division coordinates, based on the display area division coordinates and the transformation matrix described above. Specifically, the arithmetic circuit 21 calculates the image division coordinates based on the display area division coordinates and equation (2). In this specification, the image division coordinates are an example of the second division coordinates. The image division coordinates are coordinates that divide the area on the captured image corresponding to the display area 43.

[0065] The calculation circuit 21 then calculates the display device division coordinates in the coordinate system of the display device 3 that correspond to the display area division coordinates, based on the captured image division coordinates and the conversion table described above. In this specification, the display device division coordinates are an example of third division coordinates. The display device division coordinates are coordinates that divide the area on the projected content projected by the display device 3 that corresponds to the display area 43.

[0066] For example, the arithmetic circuit 21 calculates division coordinates that divide the projected content in the same way as the display area 43. The arithmetic circuit 21 calculates geometric correction information to convert each coordinate in the division coordinates of the projected content to the corresponding coordinate in the display device division coordinates. The arithmetic circuit 21 can geometrically correct the projected content using the geometric correction information.

[0067] Figure 9 is an example of an image that the arithmetic circuit 21 intends to project using the display device 3. Figure 9 shows the image 46 that the display device 3 intends to project using the optical mechanism 36 based on the image signal received from the control device 2, and the geometrically corrected projected content 47 contained within the image 46. The geometrically corrected projected content 47 shown in Figure 9 corresponds to the display area 43 shown in Figures 8A and 8B. Specifically, the geometrically corrected projected content 47 shown in Figure 9 corresponds to areas in the display area 43 shown in Figures 8A and 8B that are divided into eight horizontally and three vertically. When the display device 3 projects the image 46 onto the screen 7, the image 46 is displayed in the projection area. As shown in Figure 9, the geometrically corrected projected content 47 is located in a portion of the image 46.

[0068] The arithmetic circuit 21 generates an image signal including the geometrically corrected projection content (S18). For example, the arithmetic circuit 21 generates an image signal corresponding to the image 46 shown in Figure 9. The arithmetic circuit 21 transmits the generated image signal to the display device 3. The display device 3 projects the image 46 onto the screen 7 based on the received image signal. Because the projection content has been geometrically corrected so that it is displayed in the display area 43, when the display device 3 projects the image 46, the projection content is displayed in the area of ​​the screen 7 corresponding to the display area 43. In addition, the display device 3 projects, for example, black or a color with the minimum brightness value, for areas of the image 46 other than the geometrically corrected projection content 47. In this way, the arithmetic circuit 21 of the control device 2 can display image content in a predetermined area by having the display device 3 project the image 46 including the geometrically corrected projection content 47.

[0069] Thus, according to the display system 1 of this disclosure, the control device 2 can generate a transformation relationship that associates the coordinate system of a three-dimensional model with the coordinate system of an captured image based on the markers. Therefore, without the user having to place markers at equal intervals on the projection surface 7, the control device 2 can accurately geometrically correct the projected content and display the image content within a predetermined area through the display device 3. Consequently, the user can easily adjust the projected image through the control device 2 so that the linearity of the image content displayed in the predetermined area is ensured.

[0070] In Embodiment 1, the shape of the screen 7 is planar, but the display system 1 according to this disclosure can also be used when the shape of the screen 7 is curved. Figure 10 is another example of a display area 43 determined in a three-dimensional model 7A of the screen 7. In this example, the screen 7 is provided with three markers b1 to b3. Therefore, three markers b1 to b3 are similarly generated in the three-dimensional model 7A. The display area 43 is defined such that the three markers b1 to b3 are part of the four corners of the display area 43. As shown in Figure 10, the three-dimensional model 7A is curved in the horizontal direction. Similarly, the display area 43 is curved in the horizontal direction.

[0071] The calculation circuit 21 sets a predetermined plane 44 that intersects the display area 43 and extracts the intersection line 45 between the predetermined plane 44 and the display area 43. Then, the calculation circuit 21 calculates coordinates that equally divide the intersection line 45 and determines these coordinates as the display area division coordinates. The calculation circuit 21 can determine the display area division coordinates by calculating coordinates that divide the intersection line 45, i.e., the curve, which follows the shape of the 3D model 7A, into equal lengths. In this way, the calculation circuit 21 can determine coordinates that equally divide the display area 43 even if the shape of the display area 43 is not a plane. In the example shown in Figure 10, the predetermined plane 44 is set only in the horizontal direction, but the calculation circuit 21 may also set a predetermined plane in the vertical direction, or alternatively.

[0072] The arithmetic circuit 21 calculates division coordinates to divide the projected content in the same way as the display area 43. The arithmetic circuit 21 calculates geometric correction information to convert each coordinate in the division coordinates of the projected content to the corresponding coordinate in the display device division coordinates. In this way, the arithmetic circuit 21 can geometrically correct the projected content using the geometric correction information even if the shape of the display area 43 is not planar.

[0073] Figure 11 shows another example of an image that the display device 3 intends to project. Figure 11 shows the image 46 that the display device 3 intends to project by the optical mechanism 36, and the geometrically corrected projected content 47 contained within the image 46. The geometrically corrected projected content 47 shown in Figure 11 corresponds to the display area 43 shown in Figure 10. Specifically, the geometrically corrected projected content 47 shown in Figure 11 corresponds to areas in the display area 43 shown in Figure 10 that are divided into eight horizontally and three vertically. As shown in Figure 11, the geometrically corrected projected content 47 is located in a portion of the image 46.

[0074] As shown in Figure 11, the geometrically corrected projected content 47 is geometrically corrected so that it is concave in the vertical direction towards the center within the image 46. In this way, the shape of the geometrically corrected projected content 47 within the image 46 can change depending on the shape of the display area 43. The arithmetic circuit 21 generates an image signal including the image 46 and transmits the image signal to the display device 3. The display device 3 projects the image onto the screen 7 based on the received image signal. Since the projected content is geometrically corrected so that it is displayed in the area of ​​the screen 7 corresponding to the display area 43, when the display device 3 projects the image 46, the projected content is displayed in the display area 43. In this way, the arithmetic circuit 21 of the control device 2 can display image content in a predetermined area even if that predetermined area has a shape different from a plane, by having the display device 3 project the image 46 including the geometrically corrected projected content 47.

[0075] In the embodiment described above, the calculation circuit 21 acquires three-dimensional information, including a three-dimensional model of a predetermined space including the screen 7, from the three-dimensional shape measuring device 5. However, the calculation circuit 21 may acquire three-dimensional information from a source other than the three-dimensional shape measuring device 5. For example, the calculation circuit 21 may be configured to read and use three-dimensional information about the three-dimensional model that has been stored in the storage device 32 in advance.

[0076] In the embodiment described above, the arithmetic circuit 21 identifies the screen 7 based on the four markers a1 to a4, but the method of identifying the screen 7 is not limited to being based on the four markers. For example, the arithmetic circuit 21 may identify the surface having the markers as the screen 7. If the screen 7 is covered by a frame, the screen 7 may be identified based on the frame. Furthermore, since the arithmetic circuit 21 can generate a transformation matrix if it can obtain a set of coordinates of four points on the screen 7 and four points on the 3D model, it may proceed with processing without identifying the screen 7.

[0077] In the embodiment described above, the arithmetic circuit 21 determines the display area based on indicators b1 to b3, but the method for determining the display area is not limited to based on indicators. For example, the user may specify any three points on the three-dimensional model displayed by the output device 25 via the input device 24, and the arithmetic circuit 21 may determine the display area based on those three points.

[0078] 3. Effects According to the control device 2 or display system 1 of the embodiment of this disclosure, the following effects can be achieved.

[0079] The adjustment method is performed by a control device 2 that outputs an image signal to a display device 3 that projects an image containing image content onto a projection surface 7 in a predetermined space. The adjustment method includes acquiring three-dimensional information including a three-dimensional model of the predetermined space having a sign, and acquiring an image captured of the predetermined space having a sign from an imaging device 4. The adjustment method includes acquiring a pattern image captured of the predetermined space onto which the pattern image is projected by the display device 3 from the imaging device 4. The adjustment method includes generating a first coordinate transformation relationship that associates a first coordinate system of the three-dimensional model with a second coordinate system of the image captured based on the sign. The adjustment method includes generating a second coordinate transformation relationship that associates a second coordinate system of the pattern image captured with a third coordinate system for the projected content projected by the display device 3, based on predetermined features included in the pattern image. The adjustment method includes determining the display area of ​​the image content in the three-dimensional model based on the coordinates in the first coordinate system. The adjustment method includes generating an image signal by geometrically correcting the projected content based on a first coordinate transformation relationship and a second coordinate transformation relationship so that the display device 3 projects the image content onto the area of ​​the projection surface 7 corresponding to the display area.

[0080] According to this method, the control device 2 can generate a transformation relationship that associates the coordinate system of the three-dimensional model with the coordinate system of the captured image based on the markers. Therefore, without the user having to place markers at equal intervals on the projection surface 7, the control device 2 can accurately geometrically correct the projected content and display the image content within a predetermined area through the display device 3. Thus, the user can easily adjust the projected image through the control device 2 so that the linearity of the image content displayed in the predetermined area is ensured.

[0081] In the adjustment method, generating an image signal includes determining a first division coordinate system that evenly divides the display area in a first coordinate system. Generating an image signal includes calculating a second division coordinate system corresponding to the first division coordinate system based on the first division coordinate system and a first coordinate transformation relationship. Generating an image signal includes calculating a third division coordinate system corresponding to the second division coordinate system based on the second division coordinate system and a second coordinate transformation relationship. Generating an image signal includes geometrically correcting the projected content based on the third division coordinate system to generate an image signal.

[0082] According to this method, the control device 2 can determine coordinates that evenly divide the display area. Since the control device 2 can convert from the coordinate system of the 3D model to the coordinate system of the display device 3 using the first and second coordinate transformation relationships, it can convert the coordinates in the coordinate system of the 3D model to the coordinates in the coordinate system of the display device 3. By geometrically correcting the projected content to match these coordinates, the projected content is geometrically corrected so that it is projected within the display area, and the control device 2 can perform geometric correction with high accuracy. Therefore, the user can easily adjust the projected image through the control device 2 so that the linearity of the image content displayed in a predetermined area is ensured.

[0083] In the adjustment method, the first division coordinate is determined by equally dividing the line where the display area and a predetermined plane in the first coordinate system intersect.

[0084] With this method, the control device 2 can calculate coordinates for equally dividing the display area without the user having to specify the coordinates for equally dividing the display area. Therefore, the user can easily adjust the projected image through the control device 2 so that the linearity of the image content displayed in the predetermined area is ensured.

[0085] In the adjustment method, generating a first coordinate transformation relationship includes associating a marker on a three-dimensional model with a marker on an captured image, and generating a first coordinate transformation relationship based on the correspondence between the markers.

[0086] According to this method, the control device 2 can generate a transformation relationship that associates the coordinate system of a three-dimensional model with the coordinate system of an captured image based on markers placed in a predetermined space. Since the user does not need to place the markers at equal intervals on the projection surface 7, the user can easily adjust the projected image through the control device 2 so that the linearity of the image content displayed in a predetermined area is ensured.

[0087] In the adjustment method, the indicators are placed at the four corners of the projection surface 7.

[0088] According to this method, the control device 2 can generate a transformation relationship that associates the coordinate system of the three-dimensional model with the coordinate system of the captured image, based on markers provided at the four corners of the projection surface 7. Because the markers are provided at the four corners of the projection surface 7, this transformation relationship includes the correspondence between coordinate systems for the area where the image content is displayed. Therefore, the control device 2 can geometrically correct the projected content with high accuracy.

[0089] In the adjustment method, the display area is determined by accepting the specification of coordinates for at least three of the four corners of the display area.

[0090] According to this method, the control device 2 can determine the display area based on three specified points. Based on the first coordinate transformation relationship and the second coordinate transformation relationship, the control device 2 can identify the area in the coordinate system of the display device 3 that corresponds to the display area. By geometrically correcting the projected content to fit the area, the control device 2 can correct the projection so that the projected content is projected within the display area. Therefore, it can perform geometric correction with high accuracy without accepting user input regarding the coordinates that evenly divide the display area. Accordingly, the user can easily adjust the projected image so that the linearity of the image content displayed in the predetermined area is ensured.

[0091] In the adjustment method, three-dimensional information is obtained from the three-dimensional shape measuring device 5.

[0092] According to this method, the control device 2 can obtain a three-dimensional model of a predetermined space from the three-dimensional shape measuring device 5 to correspond with the captured image captured by the imaging device 4. Therefore, without the user having to place markers at equal intervals on the projection surface 7, the control device 2 can accurately geometrically correct the projected content and display the image content within the predetermined area through the display device 3. Thus, the user can easily adjust the projected image so that the linearity of the image content displayed in the predetermined area is ensured.

[0093] The display system 1 comprises a display device 3 that projects an image containing image content onto a projection surface 7 in a predetermined space, a control device 2 equipped with an arithmetic circuit 21 capable of outputting an image signal to the display device 3, and an imaging device 4 capable of communicating with the control device 2. The imaging device 4 is configured to capture an image of a predetermined space having a marker and generate an imaged image, and to capture a pattern image of a predetermined space onto which a pattern image has been projected by the display device 3. The arithmetic circuit 21 is configured to acquire three-dimensional information including a three-dimensional model of the predetermined space having a marker, and to acquire the imaged image and the pattern image from the imaging device 4. The arithmetic circuit 21 is configured to generate a first coordinate transformation relationship that associates a first coordinate system of the three-dimensional model with a second coordinate system of the imaged image based on the marker. The arithmetic circuit 21 is configured to generate a second coordinate transformation relationship that associates a second coordinate system of the pattern image with a third coordinate system for the projected content projected by the display device 3, based on predetermined features included in the pattern image. The arithmetic circuit 21 is configured to determine the display area of ​​the image content in the three-dimensional model based on the coordinates in the first coordinate system. The arithmetic circuit 21 is configured to generate an image signal by geometrically correcting the projected content based on the first and second coordinate transformation relationships so that the display device 3 projects the image content onto the area of ​​the projection surface 7 corresponding to the display area.

[0094] With this configuration, the control device 2 can generate a transformation relationship that associates the coordinates of the 3D model with the coordinate system of the captured image based on the markers. Therefore, without the user having to place markers at equal intervals on the projection surface 7, the control device 2 can accurately geometrically correct the projected content and display the image content within a predetermined area through the display device 3. Consequently, the display system 1 can be easily adjusted to display the image content projected by the display device 3 within a predetermined area without using evenly spaced markers.

[0095] The display system 1 further includes a three-dimensional shape measuring device that generates three-dimensional information.

[0096] With this configuration, the display system 1 can obtain a three-dimensional model of a predetermined space from the three-dimensional shape measuring device 5 to associate with the captured image captured by the imaging device 4. Therefore, the display system 1 can easily adjust the display of image content projected by the display device 3 within a predetermined area based on the signs placed in the predetermined space, the captured image, and the three-dimensional model, without using signs evenly distributed in the predetermined space.

[0097] (Embodiment 2) 1. Configuration [Display System] The display system 1 according to Embodiment 2 of the present disclosure will be described with reference to Figure 12. Figure 12 is a schematic diagram of the display system 1 according to Embodiment 2 of the present disclosure. The display system 1 comprises a control device 2, a first display device 3a, a second display device 3b, an imaging device 4, and a three-dimensional shape measuring device 5. Hereinafter in this specification, when it is not necessary to distinguish between the multiple display devices 3a and 3b, they will be collectively referred to as the display device 3.

[0098] In Embodiment 2, the control device 2 displays a portion of the image content on the screen 7 using the first display device 3a, and displays a portion of the image content on the screen 7 using the second display device 3b. The display device 3 displays a single image content by projecting an image onto the screen 7 that overlaps at least partially with the other display device 3. In Embodiment 2, the first display device 3a and the second display device 3b are arranged side by side horizontally. Therefore, the first display device 3a projects a first image containing a portion of the image content onto the left side of the screen 7. The second display device 3b projects a second image containing a portion of the image content that overlaps at least partially with the first image content onto the right side of the screen 7. The arithmetic circuit 21 generates an image signal for the first image and transmits it to the first display device 3a, causing the first display device 3a to project the first image. The arithmetic circuit 21 generates an image signal for the second image and transmits it to the second display device 3b, causing the second display device 3b to project the first image.

[0099] In Embodiment 2, the projected content projected by the first display device 3a corresponds to a portion of the image content, namely the first image content. The projected content projected by the second display device 3b corresponds to a portion of the image content, namely the second image content.

[0100] 2. Operation Below, an overview of the processing by the control device 2 of the display system 1 according to Embodiment 2 will be explained with reference to Figure 4. The arithmetic circuit 21 of the control device 2 executes the processing S10 to S11 in the same way as in Embodiment 1. In S12, the arithmetic circuit 21 acquires a pattern image for each of the multiple display devices 3. In Embodiment 2, the arithmetic circuit 21 transmits an image signal including the pattern image to the first display device 3a, causing the first display device 3a to project the pattern image. The first display device 3a projects the pattern image as projection content onto the projection area of ​​the first display device 3a. The imaging device 4 acquires the pattern image displayed on the screen 7 as the first pattern image. The arithmetic circuit 21 acquires the first pattern image from the imaging device 4.

[0101] Furthermore, the arithmetic circuit 21 transmits an image signal including the pattern image to the second display device 3b, causing the second display device 3b to project the pattern image. The second display device 3b projects the pattern image as projection content onto its projection area. The imaging device 4 acquires the pattern image displayed on the screen 7 as the second pattern image. The arithmetic circuit 21 acquires the second pattern image from the imaging device 4.

[0102] Figure 13A is an example of a first pattern image 41a captured by the imaging device 4. Figure 13B is an example of a second pattern image 41b captured by the imaging device 4. As shown in Figure 13A, the pattern image 42 is located on the left side in the first pattern image 41a. As shown in Figure 13B, the pattern image 42 is located on the right side in the second pattern image 41b. In this way, by acquiring captured images with the pattern image projected onto each display device 3, the arithmetic circuit 21 can generate a conversion table for each display device 3 that associates the coordinate system of the display device 3 with the coordinate system of the captured image in the processing of S15. The arithmetic circuit 21 may project different pattern images onto the display device 3 for the first pattern image 41a and the second pattern image 41b.

[0103] The arithmetic circuit 21 executes the processes S13 to S14 in the same manner as in Embodiment 1. In addition, as described above, in the process of S15, the arithmetic circuit 21 generates a conversion table for each display device 3 that associates the coordinate system of the display device 3 with the coordinate system of the captured image.

[0104] Next, in the processing of S16, the arithmetic circuit 21 determines the display area of ​​the image content in the 3D model. If the display area specified by the user includes at least a portion of the projection area for each of the multiple display devices 3, the arithmetic circuit 21 determines to display one image content using the multiple display devices 3.

[0105] Once the display area is determined, the arithmetic circuit 21, in the processing of S17, geometrically corrects the projected content projected by the display device 3 based on the generated transformation matrix and a plurality of transformation tables. In Embodiment 2, the arithmetic circuit 21 geometrically corrects the projected content projected by the first display device 3a using the transformation matrix and the transformation table for the first display device 3a. The arithmetic circuit 21 also geometrically corrects the projected content projected by the second display device 3b using the transformation matrix and the transformation table for the second display device 3b.

[0106] Figure 14 shows an example of an image that the arithmetic circuit 21 attempts to project using multiple display devices 3. Figure 14 shows a first image 46a that the first display device 3a attempts to project using the optical mechanism 36 based on an image signal received from the control device 2, and a second image 46b that the second display device 3b attempts to project using the optical mechanism 36 based on an image signal received from the control device 2. The first image 46a includes geometrically corrected projected content 47a projected by the first display device 3a. The second image 46b includes geometrically corrected projected content 47b projected by the second display device 3b.

[0107] In Figure 14, the first image 46a and the second image 46b are virtually positioned so that at least a portion of them overlap. The geometrically corrected projected content 47a and 47b overlap in the overlapping region 48. In Figure 14, the first image 46a and the second image 46b are described as having a positional relationship corresponding to the projection area of ​​each display device 3 on the screen 7. The first and second image content displayed on the screen 7 by projecting the geometrically corrected projected content 47a and 47b maintains the positional relationship between the geometrically corrected projected content 47a and 47b in Figure 14. Therefore, the first and second image content displayed on the screen 7 overlap in the overlapping region corresponding to the overlapping region 48.

[0108] Furthermore, the projection area of ​​the first display device 3a is offset downward relative to the projection area of ​​the second display device 3b. In this way, the display system 1 according to the present disclosure can display one image content in a predetermined area even if the projection area of ​​one of the multiple display devices 3 is offset in any direction relative to the projection area of ​​another of the multiple display devices 3.

[0109] When displaying a single image content by projecting images that overlap at least partially using multiple display devices 3, the brightness of the overlapping areas becomes higher than the brightness of the non-overlapping areas. Therefore, the arithmetic circuit 21 performs edge blending processing to adjust the brightness of the overlapping areas.

[0110] Figure 15A shows the first image 46a and the mask image 49a for the first image 46a projected by the first display device 3a. Figure 15B shows the second image 46b and the mask image 49b for the second image 46b projected by the second display device 3b. Each region of the mask image 49a corresponds to each region of the first image 46a. Each region of the mask image 49b corresponds to each region of the second image 46b.

[0111] The mask image 49a shows the brightness correction rate for the first image 46a. In the mask image 49a, areas displayed in white indicate a brightness correction rate of 100%. In other words, the calculation circuit 21 maintains the brightness of that area in the first image 46a before and after applying the mask image 49a. In the mask image, areas displayed in black indicate a brightness correction rate of 0%. In other words, the calculation circuit 21 adjusts the brightness of that area in the first image 46a to the minimum value. In the mask image 49a, areas corresponding to overlapping areas 48 indicate that the brightness correction rate changes depending on the position. In other words, the calculation circuit 21 adjusts the brightness of the overlapping areas 48 in the first image according to the position. Similarly, the mask image 49b shows the brightness correction rate for the second image 46b.

[0112] The arithmetic circuit 21 can smoothly display image content on the screen 7 by adjusting the brightness of overlapping regions 48 in the first image 46a and the second image 46b through edge blending processing. As described above, through processing S14 to S16, the arithmetic circuit 21 obtains a transformation matrix that associates the coordinate system of the 3D model with the coordinate system of the captured image, a plurality of transformation tables that associate the coordinate system of the captured image with the coordinate system of the display device, and a display area. Therefore, the arithmetic circuit 21 can determine overlapping regions in the images displayed on the screen by the plurality of display devices 3. Accordingly, the arithmetic circuit 21 can determine the region to adjust the brightness of in the edge blending processing based on the transformation matrix, the plurality of transformation tables, and the display area.

[0113] Thus, according to the display system 1 of this disclosure, even when an image is projected by multiple display devices 3, the control device 2 can display a single image content in a predetermined area through the multiple display devices 3. Furthermore, the control device 2 can geometrically correct the projected content projected by each display device 3 so that a single image content is displayed that is smoothly connected by the images projected from the multiple display devices 3, without requiring the user to place markers at equal intervals. Therefore, the user can easily adjust the projected image through the control device 2 so that the linearity of the image content displayed in the predetermined area is ensured.

[0114] 3. Effects According to the control device 2 or display system 1 of the embodiment of this disclosure, the following effects can be achieved.

[0115] The adjustment method is performed by a control device 2 that outputs an image signal to a display device 3 that projects an image containing image content onto a projection surface 7 in a predetermined space. The adjustment method includes acquiring three-dimensional information including a three-dimensional model of the predetermined space having a sign, and acquiring an image captured of the predetermined space having a sign from an imaging device 4. The adjustment method includes acquiring a pattern image captured of the predetermined space onto which the pattern image is projected by the display device 3 from the imaging device 4. The adjustment method includes generating a first coordinate transformation relationship that associates a first coordinate system of the three-dimensional model with a second coordinate system of the image captured based on the sign. The adjustment method includes generating a second coordinate transformation relationship that associates a second coordinate system of the pattern image captured with a third coordinate system for the projected content projected by the display device 3, based on predetermined features included in the pattern image. The adjustment method includes determining the display area of ​​the image content in the three-dimensional model based on the coordinates in the first coordinate system. The adjustment method includes generating an image signal by geometrically correcting the projected content based on a first coordinate transformation relationship and a second coordinate transformation relationship so that the display device 3 projects the image content onto the projection surface 7 corresponding to the display area. The display device 3 includes a first display device 3a and a second display device 3b that display the image content by projecting an image that overlaps at least a portion of the projection surface 7 in a predetermined space. The first display device 3a projects a first image containing a portion of the image content. The second display device 3b projects a second image containing a portion of the image content that overlaps at least a portion of the first image content. Acquiring a pattern image includes acquiring a first pattern image captured by the first display device 3a of the predetermined space on which the pattern image is projected, and a second pattern image captured by the second display device 3b of the predetermined space on which the pattern image is projected.Generating an image signal includes geometrically correcting the projected content of the first display device 3a and the second display device 3b based on a first coordinate transformation relationship and a second coordinate transformation relationship, so that the first display device 3a projects the first image content onto the area of ​​the projection surface 7 corresponding to the display area, and the second display device 3b projects the second image content onto the area of ​​the projection surface 7 corresponding to the display area, thereby generating a first image signal for the first image and a second image signal for the second image.

[0116] With this method, even when images are projected by multiple display devices 3, the control device 2 can display a single image content in a predetermined area through the multiple display devices 3. The control device 2 can geometrically correct the projected content projected by each display device 3 so that a single image content is displayed that connects more smoothly through the images projected from the multiple display devices 3, without requiring the user to place markers at equal intervals. Therefore, the user can easily adjust the projected image through the control device 2 so that the linearity of the image content displayed in the predetermined area is ensured.

[0117] In the adjustment method, generating the first image signal and the second image signal further includes applying edge blending to the overlapping region where the first image content and the second image content overlap to adjust the brightness of the overlapping region in the first and second images.

[0118] According to this method, the control device 2 can correct the projected content so that even in areas where images projected from multiple display devices 3 overlap, the position and brightness are smoothly connected to other areas. Therefore, the user can easily adjust the projected image through the control device 2 so that the linearity of the image content displayed in a predetermined area is ensured.

[0119] (Summary of Embodiments) As is clear from the above description, this disclosure includes the following embodiments. In the following, reference numerals are enclosed in parentheses solely to indicate the correspondence with the embodiments.

[0120] (Aspect 1) The adjustment method according to the present disclosure is an adjustment method performed by a control device (2) that outputs an image signal to a display device (3) that projects an image including image content onto a projection surface (7) of a predetermined space, and comprises: acquiring three-dimensional information including a three-dimensional model of the predetermined space having a sign; acquiring an image captured from an imaging device of the predetermined space having the sign; acquiring a pattern image captured from an imaging device of the predetermined space on which a pattern image is projected by the display device; generating a first coordinate transformation relationship that associates a first coordinate system of the three-dimensional model with a second coordinate system of the image captured based on the sign; generating a second coordinate transformation relationship that associates the second coordinate system of the pattern image captured with a third coordinate system for projected content projected by the display device based on predetermined features included in the pattern image; and determining the display area of ​​the image content in the three-dimensional model based on the coordinates in the first coordinate system. The device generates an image signal by geometrically correcting the projected content based on the first coordinate transformation relationship and the second coordinate transformation relationship so that the display device projects the image content onto the area of ​​the projection surface corresponding to the display area.

[0121] (Aspect 2) In the adjustment method of aspect 1, generating the image signal may include: determining a first division coordinate that equally divides the display area in the first coordinate system; calculating a second division coordinate in the second coordinate system corresponding to the first division coordinate based on the first division coordinate and the first coordinate transformation relationship; calculating a third division coordinate in the third coordinate system corresponding to the second division coordinate based on the second division coordinate and the second coordinate transformation relationship; and generating the image signal by geometrically correcting the projected content based on the third division coordinate.

[0122] (Aspect 3) In the adjustment method of aspect 2, the first division coordinates may be determined by equally dividing the line where the display area and a predetermined plane in the first coordinate system intersect.

[0123] (Aspect 4) In any of the adjustment methods from Aspect 1 to Aspect 3, generating the first coordinate transformation relationship may include associating the markers on the three-dimensional model with the markers on the captured image and generating the first coordinate transformation relationship based on the correspondence of the markers.

[0124] (Aspect 5) In any of the adjustment methods of aspects 1 to 4, the indicators may be provided at the four corners of the projection surface (7).

[0125] (Aspect 6) In any of the adjustment methods from aspects 1 to 5, the display area may be determined by accepting the specification of coordinates of at least three of the four corners of the display area.

[0126] (Aspect 7) In any of the adjustment methods of aspects 1 to 6, the display device (3) includes a first display device (3a) and a second display device (3b) that display the image content by projecting an image that overlaps at least a portion of the projection surface (7) of the predetermined space, the first display device projects a first image containing a portion of the image content, the second display device projects a second image containing a portion of the image content that overlaps at least a portion of the first image content, acquiring the pattern image includes acquiring a first pattern image captured by the first display device on the predetermined space onto which the pattern image is projected, and a second pattern image captured by the second display device on which the pattern image is projected, and generating the image signal, The system may also include generating a first image signal for the first image and a second image signal for the second image by geometrically correcting the projected content of the first and second display devices based on the first and second coordinate transformation relationships, such that the first display device projects the first image content onto the projection surface area corresponding to the display area, and the second display device projects the second image content onto the projection surface area corresponding to the display area.

[0127] (Aspect 8) In the adjustment method of aspect 7, generating the first image signal and the second image signal may further include applying edge blending to the overlapping region where the first image content and the second image content overlap to adjust the brightness of the overlapping region in the first image and the second image.

[0128] (Aspect 9) In any of the adjustment methods from Aspect 1 to Aspect 8, the three-dimensional information may be obtained from a three-dimensional shape measuring device (5).

[0129] (Aspect 10) The computer program relating to this disclosure causes the arithmetic circuit (21) of the computer (2) to execute any of the adjustment methods of aspects 1 to 9.

[0130] (Aspect 11) A display system (1) according to the present disclosure comprises: a display device (3) that projects an image including image content onto a projection surface (7) in a predetermined space; a control device (2) equipped with an arithmetic circuit (21) capable of outputting an image signal to the display device; and an imaging device (4) capable of communicating with the control device, wherein the imaging device is configured to capture an image of the predetermined space having a sign and generate an imaged image, and to capture a pattern image of the predetermined space onto which a pattern image has been projected by the display device, and the arithmetic circuit is configured to acquire three-dimensional information including a three-dimensional model of the predetermined space having the sign; acquire the imaged image and the pattern image from the imaging device; generate a first coordinate transformation relationship that associates the first coordinate system of the three-dimensional model with the second coordinate system of the imaged image based on the sign; and generate a second coordinate transformation relationship that associates the second coordinate system of the pattern image with the third coordinate system of the projected content projected by the display device based on predetermined features included in the pattern image. The system is configured to: determine the display area of ​​the image content in the three-dimensional model based on the coordinates in the first coordinate system; and generate the image signal by geometrically correcting the projected content based on the first coordinate transformation relationship and the second coordinate transformation relationship so that the display device projects the image content onto the area of ​​the projection surface corresponding to the display area.

[0131] (Aspect 12) The display system (1) of aspect 11 may further include a three-dimensional shape measuring device (5) that generates the three-dimensional information.

[0132] In this specification, terms such as “First,” “Second,” etc., are used for illustrative purposes only and should not be understood as expressing or implying relative importance or ranking of technical features. Features designated as “First” and “Second” express or imply that they include one or more such features.

[0133] The adjustment methods and display systems described in this disclosure are realized through the cooperation of hardware resources, such as a processor and memory, and software (computer programs).

[0134] This disclosure provides an adjustment method, a display system, and a computer program that can easily adjust images displayed in a predetermined space compared to conventional methods, and is therefore suitable for use in this type of industrial field.

Claims

1. An adjustment method performed by a control device that outputs an image signal to a display device that projects an image including image content onto a projection surface of a predetermined space, the adjustment method comprising: acquiring three-dimensional information including a three-dimensional model of the predetermined space having a marker; acquiring an image captured from an imaging device of the predetermined space having the marker; acquiring a pattern image captured from an imaging device of the predetermined space onto which a pattern image has been projected by the display device; generating a first coordinate transformation relationship that associates a first coordinate system of the three-dimensional model with a second coordinate system of the image captured based on the marker; generating a second coordinate transformation relationship that associates the second coordinate system of the pattern image captured with a third coordinate system for projected content projected by the display device based on predetermined features included in the pattern image; determining the display area of ​​the image content in the three-dimensional model based on the coordinates in the first coordinate system; and generating the image signal by geometrically correcting the projected content based on the first and second coordinate transformation relationships so that the display device projects the image content onto the area of ​​the projection surface corresponding to the display area.

2. The adjustment method according to claim 1, wherein generating the image signal includes: determining a first division coordinate that equally divides the display area in the first coordinate system; calculating a second division coordinate in the second coordinate system corresponding to the first division coordinate based on the first division coordinate and the first coordinate transformation relationship; calculating a third division coordinate in the third coordinate system corresponding to the second division coordinate based on the second division coordinate and the second coordinate transformation relationship; and generating the image signal by geometrically correcting the projected content based on the third division coordinate.

3. The adjustment method according to claim 2, wherein the first division coordinate is determined by equally dividing the line where the display area and a predetermined plane in the first coordinate system intersect.

4. The adjustment method according to claim 1, wherein generating the first coordinate transformation relationship includes associating the marker on the three-dimensional model with the marker on the captured image and generating the first coordinate transformation relationship based on the correspondence of the markers.

5. The adjustment method according to claim 1, wherein the markers are provided at the four corners of the projection surface.

6. The adjustment method according to claim 1, wherein the display area is determined by receiving the specification of coordinates of at least three of the four corners of the display area.

7. The display device includes a first display device and a second display device that display the image content by projecting an image that overlaps at least a portion of the projection surface of the predetermined space, the first display device projects a first image containing a portion of the image content, the second display device projects a second image containing a portion of the image content that overlaps at least a portion of the first image content, acquiring the pattern image includes acquiring a first pattern image captured by the first display device on the predetermined space onto which the pattern image is projected, and a second pattern image captured by the second display device on which the pattern image is projected, and generating the image signal, The adjustment method according to claim 1, comprising geometrically correcting the projected content of the first and second display devices based on the first and second coordinate transformation relationships, such that the first display device projects the first image content onto the projection surface area corresponding to the display area, and the second display device projects the second image content onto the projection surface area corresponding to the display area, thereby generating a first image signal for the first image and a second image signal for the second image.

8. The adjustment method according to claim 7, wherein generating the first image signal and the second image signal further includes applying edge blending to the overlapping region where the first image content and the second image content overlap to adjust the brightness of the overlapping region in the first image and the second image.

9. The adjustment method according to claim 1, wherein the three-dimensional information is obtained from a three-dimensional shape measuring device.

10. A computer program that causes a computer's arithmetic circuit to execute the method described in any one of claims 1 to 9.

11. A display system comprising: a display device that projects an image including image content onto a projection surface of a predetermined space; a control device equipped with an arithmetic circuit capable of outputting an image signal to the display device; and an imaging device capable of communicating with the control device, wherein the imaging device is configured to capture an image of the predetermined space having a marker and generate an imaged image, and to capture a pattern image of the predetermined space onto which a pattern image has been projected by the display device; the arithmetic circuit is configured to acquire three-dimensional information including a three-dimensional model of the predetermined space having the marker; acquire the imaged image and the pattern image from the imaging device; generate a first coordinate transformation relationship that associates a first coordinate system of the three-dimensional model with a second coordinate system of the imaged image based on the marker; generate a second coordinate transformation relationship that associates the second coordinate system of the pattern image with a third coordinate system for projected content projected by the display device based on predetermined features included in the pattern image; and determine the display area of ​​the image content in the three-dimensional model based on the coordinates in the first coordinate system. A display system configured to generate an image signal by geometrically correcting the projected content based on the first coordinate transformation relationship and the second coordinate transformation relationship so that the display device projects the image content onto the area of ​​the projection surface corresponding to the display area.

12. The system according to claim 11, further comprising a three-dimensional shape measuring device for generating the three-dimensional information.

Images