Projection control apparatus and method, and projection device and projection system

Abstract

The present application belongs to the technical field of device control, and particularly relates to a projection control apparatus and method, and a projection device and a projection system. The apparatus comprises: a display module, which is used for displaying a first interface, wherein target two-dimensional content and a target three-dimensional model are displayed in the first interface; a first detection module, which is used for detecting a first interaction instruction acting on the target two-dimensional content; a first response module, which is used for translating, in response to the first interaction instruction, the target two-dimensional content on the basis of a first feature point, so as to determine an alignment feature point in the target three-dimensional model that corresponds to the first feature point; a first determination module, which is used for determining target projection parameters of a virtual projection device on the basis of at least three target point pairs; and a communication module, which is used for sending a first instruction, wherein the first instruction instructs a physical projection device to project, on the basis of the target projection parameters, the target two-dimensional content onto a target three-dimensional object in a real scene.

Description

A projection control device, method, projection equipment, and projection system. Technical Field

[0001] This application belongs to the field of equipment control technology, and in particular relates to a projection control device, method, projection equipment and projection system. Background Technology

[0002] In fields such as modern visual arts, exhibition design, architectural visualization, and augmented reality, the precise projection of two-dimensional content (images or videos) onto three-dimensional objects has become an increasingly prevalent need. This technology not only provides users with an immersive viewing experience but also offers powerful visual support in areas such as education, training, and simulation.

[0003] When projecting two-dimensional content onto a three-dimensional object, the projected content needs to be aligned with the three-dimensional object in the real-world scene. However, the existing projection alignment process is cumbersome and not intuitive, resulting in a poor user experience. Summary of the Invention

[0004] In view of this, embodiments of this application provide a projection control device, method, projection equipment, and projection system to solve the problem that the projection alignment process in the prior art is cumbersome, not intuitive, and results in a poor user experience.

[0005] A first aspect of this application provides a projection control device, which may include:

[0006] The display module is used to display a first interface, which displays target two-dimensional content and target three-dimensional model; wherein, the target three-dimensional model is a three-dimensional model corresponding to the target three-dimensional object; and the target two-dimensional content is the content projected by the virtual projection device onto the target three-dimensional model in a virtual projection scene according to the first projection parameters.

[0007] The first detection module is used to detect the first interactive instruction applied to the target two-dimensional content;

[0008] A first response module is configured to respond to the first interaction command by translating the target two-dimensional content based on a first feature point to determine an alignment feature point in the target three-dimensional model corresponding to the first feature point; the first feature point is a marker feature point of the target two-dimensional content in the projection image of the target three-dimensional model;

[0009] The first determining module is used to determine the target projection parameters of the virtual projection device based on at least three target point pairs; wherein, the target projection parameters are the projection parameters when the target two-dimensional content is aligned with the target three-dimensional model in the virtual projection scene; any target point pair includes a first feature point and the alignment feature point corresponding to the first feature point;

[0010] A communication module is used to send a first instruction, the first instruction including the target projection parameters, the first instruction physical projection device instructing the target two-dimensional content to be projected onto the target three-dimensional object in the real scene based on the target projection parameters.

[0011] A second aspect of the embodiments of this application provides a projection control method, which may include:

[0012] The first interface displays target two-dimensional content and target three-dimensional model; wherein, the target three-dimensional model is the three-dimensional model corresponding to the target three-dimensional object; the target two-dimensional content is the content projected by the virtual projection device onto the target three-dimensional model in the virtual projection scene according to the first projection parameters;

[0013] Detect the first interactive instruction applied to the target two-dimensional content;

[0014] In response to the first interactive command, the target two-dimensional content is translated based on the first feature point to determine the alignment feature point in the target three-dimensional model corresponding to the first feature point; the first feature point is the marker feature point in the projection of the target two-dimensional content in the target three-dimensional model.

[0015] The target projection parameters of the virtual projection device are determined based on at least three target point pairs; wherein, the target projection parameters are the projection parameters when the target two-dimensional content is aligned with the target three-dimensional model in the virtual projection scene; any target point pair includes a first feature point and the alignment feature point corresponding to the first feature point;

[0016] Send a first instruction, the first instruction including the target projection parameters, the first instruction instructing a physical projection device to project the target two-dimensional content onto the target three-dimensional object in the real scene based on the target projection parameters.

[0017] A third aspect of the embodiments of this application provides a projection system, which may include:

[0018] One or more physical projection devices, and a projection control device;

[0019] The projection control device is any of the above-mentioned projection control devices, or the projection control device is used to execute any of the above-mentioned projection control methods;

[0020] Any of the aforementioned physical projection devices is used to project the target two-dimensional content onto the target three-dimensional object in the real scene based on the control of the projection control device.

[0021] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described projection control methods.

[0022] A fifth aspect of this application provides a computer program product, including a computer program that, when run, causes any of the above-described projection control methods to be executed.

[0023] A sixth aspect of this application provides a projection device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the projection device implements the steps of any of the projection control methods described above.

[0024] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment includes a display module for displaying a first interface, the first interface displaying target two-dimensional content and a target three-dimensional model; wherein, the target three-dimensional model is a three-dimensional model corresponding to a target three-dimensional object; the target two-dimensional content is the content projected onto the target three-dimensional model by a virtual projection device according to first projection parameters in a virtual projection scene; a first detection module for detecting a first interaction command acting on the target two-dimensional content; and a first response module for responding to the first interaction command by translating the target two-dimensional content based on a first feature point to determine an alignment feature point in the target three-dimensional model corresponding to the first feature point; The first feature point is a marker feature point of the target two-dimensional content in the projection image of the target three-dimensional model; the first determining module is used to determine the target projection parameters of the virtual projection device based on at least three target point pairs; wherein, the target projection parameters are the projection parameters when the target two-dimensional content is aligned with the target three-dimensional model in the virtual projection scene; any target point pair includes one first feature point and the alignment feature point corresponding to the first feature point; the communication module is used to send a first instruction, the first instruction including the target projection parameters, the first instruction instructing the physical projection device to project the target two-dimensional content onto the target three-dimensional object in the real scene based on the target projection parameters. In this embodiment, the user can move content and determine point pairs by interacting with the projection control device, and can observe the alignment effect of the two-dimensional content and the three-dimensional object in the real scene, thereby completing the projection alignment more intuitively and bringing a better user experience. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 is a schematic diagram of an application scenario in an embodiment of this application;

[0027] Figure 2 is a structural diagram of an embodiment of a projection control device according to the present application;

[0028] Figure 3 is a first schematic diagram of the feature point marking process;

[0029] Figure 4 is a second schematic diagram of the feature point marking process;

[0030] Figure 5 is a first schematic diagram of the process of translating the target two-dimensional content;

[0031] Figure 6 is a second schematic diagram of the process of translating the target two-dimensional content;

[0032] Figure 7 is a schematic diagram of the process of adjusting the external parameters of the virtual projection device;

[0033] Figure 8 is a schematic diagram for viewing the projection effect;

[0034] Figure 9 is a flowchart of an embodiment of a projection control device method according to this application;

[0035] Figure 10 is a schematic block diagram of a projection device according to an embodiment of this application. Detailed Implementation

[0036] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0038] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0039] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0040] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0041] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0042] In fields such as modern visual arts, exhibition design, architectural visualization, and augmented reality, the precise projection of two-dimensional content (images or videos) onto three-dimensional objects has become an increasingly prevalent need. This technology not only provides users with an immersive viewing experience but also offers powerful visual support in areas such as education, training, and simulation.

[0043] When projecting 2D content onto a 3D object, it is necessary to align the projected content with the 3D object in the real-world scene. Specifically, the 3D coordinates of the feature points corresponding to the 2D content in the real-world scene can be determined. Based on the 2D coordinates of each feature point (in the 2D content) and its corresponding 3D coordinates (in the real-world scene), the projection parameters (intrinsic and extrinsic parameters) of the virtual projection device when aligning the 2D content with the 3D object in the real-world scene can be determined using the Perspective-n-Point method. Based on the obtained projection parameters of the virtual projection device and the 2D content, each physical projection device can determine the projection data used for projection. According to the determined projection data, each physical projection device can accurately project the 2D content to be projected onto the 3D object in the real-world scene.

[0044] However, the projection alignment process in existing technologies is cumbersome and not intuitive, resulting in a poor user experience.

[0045] In view of this, embodiments of this application provide a projection control device, method, projection equipment, and projection system to solve the problem that the projection alignment process in the prior art is cumbersome, not intuitive, and results in a poor user experience.

[0046] The projection control device in this application embodiment can run on a physical projection device or other electronic device. For example, the projection control device in this application embodiment can be an application running on a physical projection device or other electronic device, or it can be a physical hardware device.

[0047] This application provides a projection system, including one or more physical projection devices and a projection control device.

[0048] In the projection system of this application embodiment, a projection control device can be used to control the projection of one or more physical projection devices. Specifically, the projection control device can display a projection scene, which can be referred to as a virtual projection scene. The virtual projection scene can display a target 3D model corresponding to the target 3D object in the real scene. In the virtual projection scene, the virtual projection device can project the target 2D content onto the target 3D model according to the projection parameters. If the projection parameters of the virtual projection device are adjusted, the projection image of the target 2D content in the target 3D model will also be adjusted accordingly. The projection control device can send the projection parameters of the virtual projection device and the target 2D content (via instructions) to one or more physical projection devices, and instruct the physical projection devices to use the projection parameters of the virtual projection device and the target 2D content as the projection result of the physical projection devices. The physical projection devices can project content according to the projection result. After one or more physical projection devices project according to the projection result, the projection situation in the real scene can be synchronized with the projection situation in the virtual projection scene, as shown in Figure 1, so as to achieve the effect of controlling one or more physical projection devices to project content through the projection control device.

[0049] The aforementioned target 3D object is a 3D object in a real scene used to display the projected content. Specifically, it can include, but is not limited to, any 3D object with a solid structure and a defined shape, such as buildings, exhibition halls, art models, etc.; the target 3D model is a 3D model corresponding to the target 3D object; the target 2D content is a representative image (e.g., a keyframe) of the content to be projected; and the content to be projected is an image or video that needs to be projected.

[0050] As an example, the projection control device in this application embodiment can be an electronic device, such as a computer, tablet computer, server, or mobile phone, etc., that has projection control functionality. For example, the projection control device can be a device with a display screen (also called a monitor), or it can be a device without a display screen. It is understood that if the projection control device does not have a display screen, it can be connected to an external display screen to display a virtual projection scene.

[0051] This application provides a projection device that can project content to be projected onto a target three-dimensional object in a real-world scene. As an example, the projection device may include a projection control device. For instance, the projection device may include a first projection unit and a control unit, wherein the projection control device can interact with the first projection unit as a control unit. The first projection unit is used to project the target two-dimensional content onto the target three-dimensional object in the real-world projection scene according to a first projection parameter or a target projection parameter. For ease of distinction, the projection parameter when the target two-dimensional content and the target three-dimensional object are not aligned can be referred to as the first projection parameter, and the projection parameter when the target two-dimensional content and the target three-dimensional object are aligned can be referred to as the target projection parameter. Both the first projection parameter and the target projection parameter can include extrinsic parameters and intrinsic parameters. The extrinsic parameters specifically include a rotation matrix and a translation vector, and the intrinsic parameters specifically include focal length and principal point offset. A projection control device according to an embodiment of this application will be described in detail below. Referring to Figure 2, a projection control device according to an embodiment of this application may include at least a display module 201, a first detection module 202, a first response module 203, a first determination module 204, and a communication module 205; the functions of each module are as follows:

[0052] In this embodiment of the application, the display module 201 can be used to display a first interface, which can display target two-dimensional content and target three-dimensional model.

[0053] In one specific implementation, the display module 201 can be a display screen, which can display the first interface.

[0054] Specifically, the target two-dimensional content and the target three-dimensional model can be imported into the projection control device and displayed on the first interface through the display module 201.

[0055] In one specific implementation, the target 3D model can be obtained through 3D reconstruction. For example, one or more physical projection devices can be used to scan the target 3D object to obtain scan data. Based on this scan data, the target 3D model can be constructed. When projection control of the physical projection device is required, the user can import the target 3D model into the projection control device, which can then display the target 3D model.

[0056] In this embodiment of the application, the user can select keyframes with easily distinguishable feature points from the content to be projected as the target two-dimensional content, and can import the target two-dimensional content into the projection control device.

[0057] After obtaining the target 3D model and target 2D content, the projection control device can use the display module 201 to project the target 2D content onto the target 3D model in a virtual projection scene.

[0058] The first detection module 202 in this embodiment can also be used to detect a first interaction instruction acting on the target two-dimensional content; the first interaction instruction is used to instruct the target two-dimensional content to be translated based on the first feature point to determine the alignment feature point in the target three-dimensional model corresponding to the first feature point.

[0059] As an example, the first interface can also display a movement control. The user can click the movement control to trigger the first detection module 202 to detect the first interaction command. After that, the user can use an input device such as a mouse to drag the target two-dimensional content to translate it, so as to determine the alignment feature point in the target three-dimensional model corresponding to the first feature point; or the first interaction command can also be the operation command of the user to drag the target two-dimensional content to translate it on the first interface with their finger, which can also determine the alignment feature point in the target three-dimensional model corresponding to the first feature point.

[0060] Accordingly, the first response module 203 in this embodiment can also be used to respond to the first interaction command and translate the target two-dimensional content based on the first feature point to determine the alignment feature point in the target three-dimensional model corresponding to the first feature point.

[0061] For example, the first feature point can be a preset feature point, or it can be a feature point obtained by the user from the target two-dimensional content.

[0062] In this embodiment of the application, the first determining module 204 can be used to determine the target projection parameters of the virtual projection device based on at least three target point pairs; wherein, the target projection parameters are the projection parameters of the virtual projection device when the target two-dimensional content is aligned with the target three-dimensional model in the virtual projection scene.

[0063] The communication module 205 in this embodiment can be used to send a first instruction to one or more physical projection devices. The first instruction is used to instruct the two-dimensional content of the target to be projected onto the three-dimensional object of the target in the real scene based on the target projection parameters.

[0064] This application embodiment includes a display module 201 for displaying a first interface, which displays target two-dimensional content and a target three-dimensional model; wherein, the target three-dimensional model is a three-dimensional model corresponding to a target three-dimensional object; the target two-dimensional content is the content projected by a virtual projection device onto the target three-dimensional model in a virtual projection scene according to first projection parameters; a first detection module 202 for detecting a first interactive command acting on the target two-dimensional content; a first response module 203 for responding to the first interactive command by translating the target two-dimensional content based on a first feature point to determine an alignment feature point in the target three-dimensional model corresponding to the first feature point; the first feature point is a marker feature point in the projection image of the target two-dimensional content in the target three-dimensional model; a first determination module 204 for determining the target projection parameters of the virtual projection device based on at least three target point pairs; wherein, the target projection parameters are the projection parameters when the target two-dimensional content is aligned with the target three-dimensional model in the virtual projection scene; any target point pair includes a first feature point and an alignment feature point corresponding to the first feature point; and a communication module 205 for sending a first command, which includes the target projection parameters and instructs the physical projection device to project the target two-dimensional content onto the target three-dimensional object in the real scene based on the target projection parameters. In this embodiment, users can move content and determine points by interacting with the projection control device, and can observe the alignment effect of two-dimensional content and three-dimensional objects in a real scene, thereby completing projection alignment more intuitively and bringing a better user experience.

[0065] In one possible implementation of this application, the first detection module 202 can be used to detect a third interaction instruction triggered by the user on the target two-dimensional content. The third interaction instruction is used to instruct the feature points marked by the user on the projection screen of the target two-dimensional content in the target three-dimensional model to be determined as the first feature points.

[0066] The first response module 203 in this embodiment can also be used to respond to a third interactive instruction and determine the feature points marked by the user on the projection screen of the target two-dimensional content in the target three-dimensional model as the first feature points.

[0067] Optionally, the display module 201 can also display the projection of the target two-dimensional content, including the first feature point, onto the target three-dimensional model.

[0068] In this embodiment of the application, the projection control device can also mark feature points on the target two-dimensional content based on the instructions issued by the user, so as to obtain the feature points (referred to as the first feature points) of the projected image of the target two-dimensional content marked by the user in the target three-dimensional model.

[0069] Specifically, users can select feature points (such as obvious corners, inflection points, boundary points, etc.) to be marked in the target two-dimensional content by operating the first interface. Based on the feature points selected by the user, the projection control device can generate a third interactive command. When the first detection module 202 detects the third interactive command acting on the target two-dimensional content, the first response module 203 can respond to the third interactive command and determine the feature points marked by the user on the projection screen of the target two-dimensional content in the target three-dimensional model as the first feature points.

[0070] The triggering method for the aforementioned third interactive command may include, but is not limited to, any one or more interactive methods such as mouse click, touch screen click, or keyboard tap.

[0071] For example, users can use the mouse to click on feature points of the projection of the target 2D content onto the target 3D model on the first interface, as shown in Figure 3, to trigger a third interactive command to mark the feature point.

[0072] For example, users can tap on feature points of the projection of the target 2D content onto the target 3D model on the first interface, as shown in Figure 4, to trigger a third interactive command to mark the feature point.

[0073] Specifically, after determining the first feature point, the user can move (translate) the projection of the target 2D content onto the target 3D model in the first interface, so that the first feature point in the projection is aligned (coincidentally) with the corresponding feature point on the target 3D model. Based on the user's translation of the target 2D content, the projection control device can generate a first interaction command. When the first detection module 202 detects the first interaction command triggered by the user on the target 2D content, the first response module 203 can respond to the first interaction command and translate the target 2D content based on the first feature point to determine the aligned feature point on the target 3D model corresponding to the first feature point.

[0074] The triggering method of the aforementioned first interactive command may include, but is not limited to, any one or more interactive methods such as mouse dragging, touch screen dragging, or keyboard typing.

[0075] For example, a user can click on the first feature point on the first interface with the mouse. Then, the user can press and hold the first feature point with the mouse and drag it to pan the projection of the target two-dimensional content in the target three-dimensional model, as shown in Figure 5.

[0076] For example, users can tap the first feature point on the first interface via touchscreen and drag the first feature point to translate the projection of the target two-dimensional content onto the target three-dimensional model, as shown in Figure 6.

[0077] It is important to understand that in a virtual projection scenario, the virtual projection device can project target 2D content onto the target 3D model based on its own projection parameters. If the extrinsic parameters of the virtual projection device change, the position of the target 2D content on the projected image of the target 3D model will also change accordingly, as shown in Figure 7. Therefore, in this embodiment, the translation of the target 2D content can be achieved by adjusting the extrinsic parameters of the virtual projection device. Specifically, based on the user's translation of the target 2D content, the position of the translated target 2D content on the projected image of the target 3D model can be determined. Based on the position of the translated target 2D content on the projected image of the target 3D model, the first projection parameters of the translated virtual projection device can be determined. According to these first projection parameters, the projection module can be used to control the virtual projection device to project the target 2D content onto the target 3D model in the virtual projection scenario, thereby achieving the translation of the target 2D content in the first interface.

[0078] After translating the projection of the target 2D content onto the target 3D model, the projection control device can determine the alignment feature point in the target 3D model corresponding to the first feature point, and display the alignment feature point in the target 3D model on the first interface. Since the target 3D model corresponds to the target 3D object in the real scene, the coordinates of the alignment feature point in the target 3D model are the same as the coordinates of the first feature point in the real scene.

[0079] After translating the projection of the target 2D content onto the target 3D model, the user can further determine the first feature point and its corresponding alignment feature point. Here, the user can determine the first feature point and alignment feature point using preset controls in the first interface; for example, the user can use a preset button in the first interface displaying "Determine Target Point Pair" to determine the first feature point and alignment feature point as a target point pair. When the user performs the relevant operation to determine the target point pair, a fourth interactive command can be triggered, which can be used to instruct the user to determine the first feature point and its corresponding alignment feature point as a target point pair.

[0080] Specifically, the first detection module 202 in the projection control device of this application embodiment can also be used to detect the fourth interaction command triggered by the user on the target two-dimensional content. Correspondingly, the first response module 203 can also be used to respond to the fourth interaction command and determine the first feature point and the alignment feature point corresponding to the first feature point as a target point pair.

[0081] Optionally, the user can also adjust the first feature point and / or the alignment feature point in the target point pair according to the actual alignment of the target point pair. For example, the user can trigger a second interactive command through a mouse, touch screen, or keyboard key to fine-tune any of the first feature points and / or any of the alignment feature points.

[0082] Accordingly, the first detection module 202 can also be used to detect a second interaction command triggered by the user on the first interface. The second interaction command can be used to instruct the adjustment of the position of any first feature point, and / or the adjustment of the position of any aligned feature point. The first response module 203 can also be used to adjust the position of any first feature point, and / or the adjustment of the position of any aligned feature point in response to the second interaction command. Afterwards, the adjusted first feature point, and / or the adjusted aligned feature point, can be determined as the adjusted target point pair.

[0083] It should be noted that the communication module 205 in the projection control device can send a second instruction to one or more physical projection devices in the projection system in real time. The second instruction may include the first projection parameters and the target two-dimensional content. The second instruction can be used to instruct the target two-dimensional content to be projected onto the target three-dimensional object in the real scene based on the first projection parameters. After receiving the first projection parameters and the target two-dimensional content, the physical projection device can determine the projection data of the physical projection device. Based on the projection data, the physical projection device can project the target two-dimensional content onto the target three-dimensional object in the real scene so that the projection situation in the real scene is synchronized with the projection situation in the virtual projection scene.

[0084] By repeating the above process, the user can determine at least three target point pairs for subsequent n-point perspective pose solving.

[0085] In this embodiment, the target projection parameters can be solved using an n-point perspective pose solving method. Specifically, based on the first projection parameters, aligned feature points can be reprojected onto the image coordinate system (of the target's two-dimensional content); then, the error of this reprojection process can be used as an optimization objective to solve for the target projection parameters.

[0086] Specifically, the first determining module 204 in the projection control device may include a reprojection submodule, an error calculation submodule, and an optimization solution submodule.

[0087] The reprojection submodule can be used to reproject each aligned feature point into the image coordinate system based on the first projection parameters to obtain the corresponding reprojected feature points.

[0088] It is important to understand that in actual projection, two-dimensional feature points can be transformed from the image coordinate system to the projector coordinate system based on the intrinsic parameters of the projection device. Then, based on the extrinsic parameters of the projection device, the feature points obtained in the projector coordinate system can be transformed to the coordinate system of the virtual projection scene, thus obtaining the three-dimensional feature points corresponding to the two-dimensional feature points in the coordinate system of the virtual projection scene.

[0089] In this embodiment of the application, the reverse process described above can be used to reproject the obtained three-dimensional feature points onto the image coordinate system to obtain the reprojected two-dimensional feature points. Here, the difference between the reprojected feature points and the original two-dimensional feature points can be calculated by reprojecting the aligned feature points onto the image coordinate system, and the projection parameters of the virtual projection device that minimize the difference can be solved.

[0090] After reprojecting the aligned feature points into the image coordinate system, the error calculation submodule can be used to calculate the reprojection error based on the reprojected feature points and the marked feature points. Specifically, the difference between the reprojected feature points and the corresponding first feature points can be calculated, and this difference is the reprojection error.

[0091] Then, the optimization solution submodule can be used to solve for the target projection parameters by using the reprojection error as the optimization objective.

[0092] In this embodiment of the application, the corresponding constraints can be flexibly set according to the number of target point pairs, and the target projection parameters can be solved based on the constraints.

[0093] Specifically, if the number of target point pairs is less than or equal to 4, the principal point offset in the target projection parameters can be set to a fixed value. The specific value of this fixed value can be set according to actual needs, and this embodiment does not limit it. If the number of target point pairs is less than 6, the ratio of the horizontal axis (x-axis) component to the vertical axis (y-axis) component of the focal length in the target projection parameters can be set to a fixed value. The specific value of this fixed value can be set according to actual needs, and this embodiment does not limit it.

[0094] Furthermore, when solving for the target projection parameters, the depth component of the translation vector in the target projection parameters can be constrained to a value less than a depth threshold to avoid the virtual projection device's pose being at infinity. After obtaining the target projection parameters, a first instruction can be sent to one or more physical projection devices via the communication module 205. This first instruction can specifically include the target projection parameters and the target two-dimensional content. The first instruction can be used to instruct the physical projection devices to project the target two-dimensional content onto the target three-dimensional object in the real scene based on the target projection parameters. Thus, the user can intuitively view the projection effect on the first interface and in the real scene, as shown in Figure 8.

[0095] If the target 2D content and the target 3D object are not aligned, the user can trigger the first operation command to make the projection control device determine that the target 2D content and the target 3D object are not aligned; if the target 2D content and the target 3D object are aligned, the user can trigger the second operation command to make the projection control device determine that the target 2D content and the target 3D object are aligned, and can save each target point pair and the target projection parameters as the alignment result, which can be used for projection in subsequent projections.

[0096] Specifically, the projection control device may also include a second detection module and a second response module.

[0097] The second detection module can be used to detect the first or second operation command input by the user; specifically, the first operation command can be used to indicate that the target two-dimensional content and the target three-dimensional object are not aligned, and the second operation command can be used to indicate that the target two-dimensional content and the target three-dimensional object are aligned.

[0098] Correspondingly, the second response module can be used to respond to the first operation instruction and determine that when the physical projection device projects the target two-dimensional content onto the target three-dimensional object in the real scene according to the target projection parameters, the target two-dimensional content and the target three-dimensional object are not aligned; or, the second response module can also be used to respond to the second operation instruction and save each target point pair and the target projection parameters as the alignment result.

[0099] If it is determined that the target 2D content and the target 3D object are not aligned when the physical projection device projects the target 2D content onto the target 3D object in the real scene according to the target projection parameters, the user can add, delete or modify the target point pairs to redetermine at least three target point pairs; then, a new round of target projection parameter calculation can be performed based on the redetermined at least three target point pairs.

[0100] After obtaining the alignment result, one or more physical projection devices in the projection system can project the content to be projected onto the target 3D object in the real scene according to the saved target projection parameters and the content to be projected.

[0101] Corresponding to the functions of each module of the aforementioned projection control device, this application provides a projection control method; specifically, please refer to Figure 9. An embodiment of a projection control method applied to a projection control device in this application may include steps S901 to S905:

[0102] Step S901: Display the first interface.

[0103] The first interface displays the target 2D content and the target 3D model; the target 3D model is the 3D model corresponding to the target 3D object; the target 2D content is the content projected by the virtual projection device onto the target 3D model in the virtual projection scene according to the first projection parameters.

[0104] In this embodiment of the application, step S901 can be executed by the display module 201 in the projection control device. For details, please refer to the relevant description above, which will not be repeated here.

[0105] In this embodiment, the virtual projection device can also be controlled to project the target two-dimensional content onto the target three-dimensional model in the virtual projection scene according to the first projection parameters or the target projection parameters; and a second instruction can also be sent to one or more physical projection devices, wherein the second instruction may include the first projection parameters, and the second instruction can be used to instruct the target two-dimensional content to be projected onto the target three-dimensional object in the real scene based on the first projection parameters; after receiving the second instruction, the physical projection device can determine the projection result based on the projection parameters (first projection parameters or target projection parameters) and the target two-dimensional content, and can project the target two-dimensional content onto the target three-dimensional object in the real scene based on the projection result.

[0106] The projection process described above can be executed by the projection module in the projection control device, and the process of sending the second instruction to the physical projection device can be executed by the communication module 205 in the projection control device. For details, please refer to the relevant description above, which will not be repeated here.

[0107] Step S902: Detect the first interactive instruction applied to the target two-dimensional content.

[0108] Step S903: In response to the first interactive instruction, translate the target two-dimensional content based on the first feature point to determine the alignment feature point in the target three-dimensional model corresponding to the first feature point.

[0109] Among them, the first feature point is the marked feature point of the target two-dimensional content in the projection image of the target three-dimensional model.

[0110] In this embodiment of the application, step S902 can be executed by the first detection module 202 in the projection control device, and step S903 can be executed by the first response module 203 in the projection control device. For details, please refer to the relevant description above, which will not be repeated here.

[0111] In this embodiment of the application, a third interaction command triggered by the user on the target two-dimensional content can also be detected. If the third interaction command is detected, the feature point marked by the user on the projection screen of the target two-dimensional content in the target three-dimensional model can be determined as the first feature point in response to the third interaction command.

[0112] The process of detecting the third interactive command described above can be executed by the first detection module 202 in the projection control device, and the process of responding to the second interactive command can be executed by the first response module 203 in the projection control device. For details, please refer to the relevant description above, which will not be repeated here.

[0113] In this embodiment of the application, a second interaction instruction triggered by the user on the first interface can also be detected. The second interaction instruction can be used to instruct the adjustment of any first feature point and / or the adjustment of any alignment feature point. After the second interaction instruction is detected, the first feature point can be adjusted and / or the alignment feature point can be adjusted in response to the second interaction instruction. Correspondingly, any target point pair also includes the adjusted first feature point and / or the adjusted alignment feature point.

[0114] The process of detecting the second interactive command described above can be executed by the first detection module 202 in the projection control device, and the process of responding to the second interactive command can be executed by the first response module 203 in the projection control device. For details, please refer to the relevant description above, which will not be repeated here.

[0115] Step S904: Determine the target projection parameters of the virtual projection device based on at least three target point pairs.

[0116] The target projection parameters are the projection parameters when the two-dimensional content of the target is aligned with the three-dimensional model of the target in the virtual projection scene; any pair of target points includes a first feature point and an alignment feature point corresponding to the first feature point.

[0117] In this embodiment of the application, step S904 can be executed by the first determining module 204 in the projection control device. For details, please refer to the relevant description above, which will not be repeated here.

[0118] Step S905: Send the first instruction.

[0119] The first instruction includes target projection parameters, which instruct the physical projection device to project the two-dimensional content of the target onto the three-dimensional object in the real scene based on the target projection parameters.

[0120] In this embodiment of the application, step S905 can be executed by the communication module 205 in the projection control device. For details, please refer to the relevant description above, which will not be repeated here.

[0121] In addition, after calculating the target projection parameters, the user can determine whether the target 2D content is aligned with the target 3D object. If it is determined that the target 2D content is not aligned with the target 3D object, the user can input a first operation command to the projection control device, causing the projection control device to determine that the target 2D content is not aligned with the target 3D object, and can then re-select the target point pair and determine the target projection parameters by referring to the above process. If it is determined that the target 2D content is aligned with the target 3D object, the user can input a second operation command to cause the projection control device to determine that the target 2D content is not aligned with the target 3D object.

[0122] Specifically, the detection process of the first operation command or the second operation command can be executed by the first detection module 202 in the projection control device, and the process of determining that the target two-dimensional content and the target three-dimensional object are not aligned / aligned can be executed by the second response module. For details, please refer to the relevant description above, which will not be repeated here.

[0123] In summary, the embodiments of this application include a display module for displaying a first interface, the first interface displaying target two-dimensional content and a target three-dimensional model; wherein, the target three-dimensional model is a three-dimensional model corresponding to a target three-dimensional object; the target two-dimensional content is the content projected by a virtual projection device onto the target three-dimensional model in a virtual projection scene according to first projection parameters; a first detection module for detecting a first interaction command triggered by a user on the target two-dimensional content; and a first response module for responding to the first interaction command by translating the target two-dimensional content based on a first feature point to determine an alignment feature point in the target three-dimensional model corresponding to the first feature point; the first feature point is the target object marked by the user. The system identifies feature points of the two-dimensional content projected onto the target three-dimensional model; a first determining module 204 is used to determine target projection parameters of the virtual projection device based on at least three target point pairs; wherein the target projection parameters are the projection parameters when the target two-dimensional content is aligned with the target three-dimensional model in the virtual projection scene; each target point pair includes a first feature point and an alignment feature point corresponding to the first feature point; a communication module is used to send a first instruction to one or more physical projection devices, the first instruction including the target projection parameters, the first instruction being used to instruct the target two-dimensional content to be projected onto the target three-dimensional object in the real scene based on the target projection parameters. In this embodiment, the user can move content and determine point pairs by interacting with the projection control device, and can observe the alignment effect of the two-dimensional content and the three-dimensional object in the real scene, thereby completing the projection alignment more intuitively and bringing a better user experience.

[0124] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0126] Figure 10 shows a schematic block diagram of a projection device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0127] As shown in FIG10, the projection device 10 of this embodiment includes: a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100. When the processor 100 executes the computer program 102, it implements the steps in the various projection alignment method embodiments described above, such as steps S901 to S905 shown in FIG9. Alternatively, when the processor 100 executes the computer program 102, it implements the functions of each module / unit in the various device embodiments described above, such as the functions of modules 201 to 205 shown in FIG2.

[0128] For example, the computer program 102 may be divided into one or more modules / units, which are stored in the memory 101 and executed by the processor 100 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 102 in the projection device 10.

[0129] Those skilled in the art will understand that FIG10 is merely an example of the projection device 10 and does not constitute a limitation on the projection device 10. It may include more or fewer components than shown, or combine certain components, or different components. For example, the projection device 10 may also include input / output devices, network access devices, buses, etc.

[0130] The processor 100 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0131] The memory 101 can be an internal storage unit of the projection device 10, such as a hard disk or memory of the projection device 10. The memory 101 can also be an external storage device of the projection device 10, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the projection device 10. Furthermore, the memory 101 can include both internal and external storage units of the projection device 10. The memory 101 is used to store the computer program and other programs and data required by the projection device 10. The memory 101 can also be used to temporarily store data that has been output or will be output.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0133] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0134] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] In the embodiments provided in this application, it should be understood that the disclosed apparatus / projection device and method can be implemented in other ways. For example, the apparatus / projection device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0138] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0139] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A projection control device, characterized in that, include: The display module is used to display a first interface, which displays target two-dimensional content and target three-dimensional model; wherein, the target three-dimensional model is a three-dimensional model corresponding to the target three-dimensional object; and the target two-dimensional content is the content projected by the virtual projection device onto the target three-dimensional model in a virtual projection scene according to the first projection parameters. The first detection module is used to detect the first interactive instruction applied to the target two-dimensional content; A first response module is configured to respond to the first interaction command by translating the target two-dimensional content based on a first feature point to determine an alignment feature point in the target three-dimensional model corresponding to the first feature point; the first feature point is a marker feature point of the target two-dimensional content in the projection image of the target three-dimensional model; The first determining module is used to determine the target projection parameters of the virtual projection device based on at least three target point pairs; wherein, the target projection parameters are the projection parameters when the target two-dimensional content is aligned with the target three-dimensional model in the virtual projection scene; any target point pair includes a first feature point and the alignment feature point corresponding to the first feature point; A communication module is used to send a first instruction, the first instruction including the target projection parameters, the first instruction instructing a physical projection device to project the target two-dimensional content onto the target three-dimensional object in the real scene based on the target projection parameters.

2. The projection control device according to claim 1, characterized in that, The first detection module is further configured to detect a second interaction command triggered by the user on the first interface, wherein the second interaction command indicates adjusting the position of any of the first feature points, and / or adjusting the position of any of the alignment feature points; The first response module is further configured to, in response to the second interaction instruction, adjust the position of any of the first feature points, and / or adjust the position of any of the aligned feature points; Accordingly, the first determining module is specifically used to determine the target projection parameters of the virtual projection device based on any of the adjusted first feature points included in any of the at least three target point pairs, and / or any of the adjusted alignment feature points.

3. The projection control device according to claim 1, characterized in that, The first detection module is also used to detect a third interactive command triggered by the user on the target two-dimensional content; The first response module is further configured to, in response to the third interaction instruction, determine the feature points marked by the user on the projection screen of the target two-dimensional content in the target three-dimensional model as the first feature points.

4. The projection control device according to claim 1, characterized in that, The projection control device further includes: The projection module is used to control the virtual projection device to project the target two-dimensional content onto the target three-dimensional model in the virtual projection scene according to the first projection parameters or the target projection parameters.

5. The projection control device according to any one of claims 1 to 4, characterized in that, The communication module is further configured to send a second instruction, the second instruction including the first projection parameters, the second instruction being configured to instruct the physical projection device to project the target two-dimensional content onto the target three-dimensional object in the real scene based on the first projection parameters.

6. The projection control device according to any one of claims 1 to 4, characterized in that, The projection control device further includes: The second detection module is used to detect a first operation command or a second operation command, wherein the first operation command indicates that the target two-dimensional content and the target three-dimensional object are not aligned, and the second operation command indicates that the target two-dimensional content and the target three-dimensional object are aligned; The second response module is configured to, in response to the first operation instruction, determine that when the physical projection device projects the target two-dimensional content onto the target three-dimensional object in the real scene according to the target projection parameters, the target two-dimensional content and the target three-dimensional object are not aligned; or, in response to the second operation instruction, save each of the target point pairs and the target projection parameters as an alignment result.

7. The projection control device according to any one of claims 1 to 4, characterized in that, The first determining module includes: The reprojection submodule is used to reproject each of the aligned feature points into the image coordinate system based on the first projection parameters to obtain the corresponding reprojected feature points. The error calculation submodule is used to calculate the corresponding reprojection error based on each of the reprojection feature points and the first feature points corresponding to each of the reprojection feature points. An optimization solution submodule is used to solve for the target projection parameters by taking each of the reprojection errors as the optimization objective.

8. A projection control method, characterized in that, include: The first interface displays target two-dimensional content and target three-dimensional model; wherein, the target three-dimensional model is the three-dimensional model corresponding to the target three-dimensional object; the target two-dimensional content is the content projected by the virtual projection device onto the target three-dimensional model in the virtual projection scene according to the first projection parameters; Detect the first interactive instruction applied to the target two-dimensional content; In response to the first interactive command, the target two-dimensional content is translated based on the first feature point to determine the alignment feature point in the target three-dimensional model corresponding to the first feature point; the first feature point is the marker feature point in the projection of the target two-dimensional content in the target three-dimensional model. The target projection parameters of the virtual projection device are determined based on at least three target point pairs; wherein, the target projection parameters are the projection parameters when the target two-dimensional content is aligned with the target three-dimensional model in the virtual projection scene; any target point pair includes a first feature point and the alignment feature point corresponding to the first feature point; Send a first instruction, the first instruction including the target projection parameters, the first instruction instructing a physical projection device to project the target two-dimensional content onto the target three-dimensional object in the real scene based on the target projection parameters.

9. The projection control method according to claim 8, characterized in that, Also includes: Detect a second interaction command triggered by the user on the first interface, the second interaction command instructing adjustment of the position of any of the first feature points, and / or adjustment of the position of any of the alignment feature points; In response to the second interaction command, adjust the position of any of the first feature points, and / or adjust the position of any of the aligned feature points; The determination of the target projection parameters of the virtual projection device based on at least three target point pairs includes: The target projection parameters of the virtual projection device are determined based on any of the adjusted first feature points included in any of the at least three target point pairs, and / or any of the adjusted alignment feature points.

10. The projection control method according to claim 8, characterized in that, Also includes: Detect the third interaction command triggered by the user on the target two-dimensional content; In response to the third interactive instruction, the feature point marked by the user on the projection screen of the target two-dimensional content in the target three-dimensional model is determined as the first feature point.

11. The projection control method according to claim 8, characterized in that, The method further includes: The virtual projection device is controlled to project the target two-dimensional content onto the target three-dimensional model in the virtual projection scene according to the first projection parameters or the target projection parameters.

12. The projection control method according to any one of claims 8 to 11, characterized in that, Also includes: Send a second instruction, the second instruction including the first projection parameters, the second instruction being used to instruct the physical projection device to project the target two-dimensional content onto the target three-dimensional object in the real scene based on the first projection parameters.

13. The projection control method according to any one of claims 8 to 11, characterized in that, The method further includes: Detect a first operation command or a second operation command, wherein the first operation command indicates that the target two-dimensional content and the target three-dimensional object are not aligned, and the second operation command indicates that the target two-dimensional content and the target three-dimensional object are aligned; In response to the first operation instruction, it is determined that when the physical projection device projects the target two-dimensional content onto the target three-dimensional object in the real scene according to the target projection parameters, the target two-dimensional content and the target three-dimensional object are not aligned; or, in response to the second operation instruction, it is used to save each of the target point pairs and the target projection parameters as an alignment result.

14. The projection control method according to any one of claims 8 to 11, characterized in that, The determination of the target projection parameters of the virtual projection device based on at least three target point pairs includes: Based on the first projection parameters, each of the aligned feature points is reprojected onto the image coordinate system to obtain the corresponding reprojected feature points; Based on each of the reprojection feature points and the marked feature points corresponding to each of the reprojection feature points, the corresponding reprojection error is calculated. The target projection parameters are solved by using each of the reprojection errors as optimization objectives.

15. A projection system, characterized in that, include: One or more physical projection devices, and a projection control device; the projection control device is the projection control device according to any one of claims 1 to 7, or the projection control device is used to perform the method according to any one of claims 8 to 14; Any of the aforementioned physical projection devices is used to project the target two-dimensional content onto the target three-dimensional object in the real scene based on the control of the projection control device.

16. A projection device, characterized in that, The projection device includes a projection control device as described in any one of claims 1 to 7, or the projection device is used to perform a projection control method as described in any one of claims 8 to 14.

Images