Method and system for integrated control of indoor infrastructure devices by using markers
The marker-based calibration method addresses the inefficiencies of manual infrastructure device control by providing accurate, cost-effective, and scalable control integration, allowing rapid recalibration.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- ADVANCED INST OF CONVERGENCE TECH
- Filing Date
- 2025-10-22
- Publication Date
- 2026-07-02
Smart Images
Figure KR2025016791_02072026_PF_FP_ABST
Abstract
Description
Method and system for integrated control of indoor infrastructure devices using markers
[0001] The present invention relates to a method and system for integrated control of indoor infrastructure devices using a marker, and more specifically, to a method and system for calculating a position and attitude value of a marker through an external calibration process, calculating a coordinate transformation relationship between a marker and an infrastructure device based on the position and attitude value, and integrally controlling multiple infrastructure devices based on the calculated coordinate transformation relationship.
[0002] Conventional infrastructure device control technology required manual human setting of the position and angle of devices such as stage lighting, broadcast cameras, and HVAC systems. Consequently, large-scale installations demanded significant human resources, increased the likelihood of errors during the setting process, and extended working times.
[0003] In addition, if the installed device changes location or environmental conditions change, the existing settings become invalid, causing the inconvenience of having to remeasure and reset the device's position and orientation.
[0004] As such, existing infrastructure device control technology required a significant amount of time and cost for the initial setup and adjustment of individual devices.
[0005] Accordingly, there is a need for a method to integrally control multiple infrastructure devices within an indoor environment by utilizing external calibration techniques.
[0006] The present disclosure aims to solve the problems of the aforementioned prior art by providing a method and system for calculating a position and attitude value of a marker through an external calibration process, calculating a coordinate transformation relationship between a marker and an infrastructure device based on the position and attitude value, and integrally controlling multiple infrastructure devices based on the calculated coordinate transformation relationship.
[0007] The technical problems that the present invention aims to solve are not limited to the technical problems described above, and other technical problems of the present invention may be derived from the following description.
[0008] As a technical means for solving the aforementioned technical problem, an embodiment according to the first aspect of the present disclosure provides an integrated control method for an infrastructure device. The method comprises the steps of: receiving from a sensor first positional relationship information between a marker and a sensor based on calibration between a marker placed in an arbitrary area of a specific space and a sensor that recognizes the marker; deriving third positional relationship information between the infrastructure device and the marker based on the first positional relationship information and a second positional relationship information between the infrastructure device placed in an area outside the region where the sensor and the marker are located within the specific space; and, when receiving a control input that causes the infrastructure device to perform an operation regarding a target area within the specific space, deriving fourth positional relationship information between the target area and the infrastructure device based on the coordinates of the target area and the third positional relationship information.
[0009] As a technical means for solving the technical problem described above, an embodiment according to a second aspect of the present disclosure provides an infrastructure device integrated control device. The device comprises a communication module, at least one processor, and a memory electrically connected to the processor and storing at least one code executed by the processor. The memory stores a code that, when executed through the processor, causes the processor to receive first positional relationship information between the marker and the sensor based on calibration between the marker placed in an arbitrary area of a specific space and the sensor recognizing the marker, and to derive a third positional relationship information between the infrastructure device and the marker based on the first positional relationship information and a pre-set second positional relationship information between the infrastructure device placed in an area outside the area where the sensor and the marker are located within the specific space, and to derive a fourth positional relationship information between the target area and the infrastructure device based on the coordinates of the target area and the third positional relationship information when receiving a control input that causes the infrastructure device to perform an operation on a target area within the specific space.
[0010] According to the present invention, accurate control with a lower error rate can be achieved through a marker-based calibration technique.
[0011] In addition, according to the present invention, the time and cost required during the initial setup and maintenance process can be significantly reduced.
[0012] In addition, according to the present invention, it can be applied to various indoor infrastructure devices such as stage lighting, broadcasting equipment, and air conditioning systems, and can provide scalability that allows it to be integrated into existing devices.
[0013] In addition, according to the present invention, rapid resetting is possible through a simple calibration process in response to environmental changes, and control can be maintained even when the sensor is removed.
[0014] The effects of the present invention are not limited to the effects described above, but include all effects understood from the following description.
[0015] FIG. 1 is a drawing illustrating a server and external devices connected to it in communication according to an embodiment of the present invention.
[0016] Figure 2 is a diagram illustrating the detailed configuration of the server shown in Figure 1.
[0017] FIG. 3 is a diagram illustrating an example for calculating the coordinate transformation relationship between a sensor and a marker according to an embodiment of the present invention.
[0018] FIG. 4 is a diagram illustrating an example for deriving fourth position relationship information of an infrastructure device according to an embodiment of the present invention.
[0019] FIG. 5 is a diagram illustrating an example of controlling an infrastructure device according to an embodiment of the present invention.
[0020] FIG. 6 is a flowchart illustrating the sequence of an infrastructure device integrated control method according to another embodiment of the present invention.
[0021] The present disclosure will be described in detail below with reference to the attached drawings. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. Furthermore, the attached drawings are intended only to facilitate understanding of the embodiments disclosed herein, and the technical concept disclosed herein is not limited by the attached drawings. All terms used herein, including technical and scientific terms, should be interpreted in the sense generally understood by those skilled in the art to which the present disclosure pertains. Terms defined in advance should be interpreted as having additional meanings consistent with relevant technical literature and the present disclosure, and should not be interpreted in a highly ideal or restrictive sense unless otherwise defined.
[0022] In order to clearly explain the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and the size, form, and shape of each component shown in the drawings may be varied. Throughout the specification, identical or similar parts are denoted by identical or similar reference numerals.
[0023] In the following description, suffixes such as "module" and "part" for components are assigned or used interchangeably solely for the ease of drafting the specification, and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, detailed descriptions of related prior art have been omitted where it is determined that such detailed descriptions could obscure the essence of the embodiments disclosed in this specification.
[0024] Throughout the specification, when it is stated that a part is "connected (connected, contacted, or coupled)" to another part, this includes not only cases where they are "directly connected (connected, contacted, or coupled)," but also cases where they are "indirectly connected (connected, contacted, or coupled)" with other members interposed therebetween. Furthermore, when it is stated that a part "includes (provides, or provides)" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but rather allows for additional "included (provided, or provided)" of other components.
[0025] Terms indicating ordinal numbers, such as first, second, etc., used herein are used solely for the purpose of distinguishing one component from another and do not limit the order or relationship of the components. For example, the first component of the present disclosure may be named the second component, and similarly, the second component may be named the first component. Singular forms used herein should be interpreted to include plural forms unless explicitly to the contrary.
[0026] FIG. 1 is a drawing illustrating a server and external devices connected to it in communication according to an embodiment of the present invention.
[0027] Referring to FIG. 1, the infrastructure device integrated control system may include a server (100), a sensor (200), a marker (300), and an infrastructure device (400).
[0028] The server (100) can be connected to communicate with at least one of the sensor (200), marker (300), and infrastructure device (400) through a pre-configured network.
[0029] The server (100) may be an infrastructure device integrated control device.
[0030] The server (100) receives from the sensor first position relationship information between the marker and the sensor according to calibration between the marker placed in an arbitrary area of a specific space and the sensor that recognizes the marker.
[0031] The server (100) derives third position relationship information between the infrastructure device and the marker based on the first position relationship information and the second position relationship information between the infrastructure device and the sensor, which are placed in an area other than the area where the sensor and the marker are located within a specific space.
[0032] When the server (100) receives a control input that causes the infrastructure device to perform an operation on a target area within the specific space, it derives a fourth positional relationship information between the target area and the infrastructure device based on the coordinates of the target area and the third positional relationship information.
[0033] The server (100) can generate a control signal for controlling an infrastructure device based on fourth position relationship information. Here, the control signal may be an operation position and a direction value.
[0034] The sensor (200) may include various types of sensors used to recognize the marker (300). For example, an image of the marker may be captured through a camera, which is a capturing device, and the physical distance to the marker may be measured using a distance sensor. A LiDAR sensor may be used to scan the surrounding environment in three dimensions to confirm the presence of the marker or to determine its location more accurately. The sensor (200) may recognize the visual or spatial features of the marker.
[0035] The sensor (200) can be placed at a location where the marker (300) comes into the field of view.
[0036] The marker (300) may be a primary object that provides a reference point during the external calibration process.
[0037] The marker (300) may be a unique pattern or a fixed structural shape. For example, the marker may be a QR code or an Aruco marker.
[0038] The marker (300) is fixedly placed in a specific space and can serve as a reference for the operation of other infrastructure devices (400) within that space. For example, the marker (300) is located in a space where the infrastructure device (400) is intended to operate and can serve as a reference for the operation of other infrastructure devices (400) within that space.
[0039] The infrastructure device (400) is a variety of physical devices requiring control in a specific environment, and may be at least one. For example, the infrastructure device (400) is a mechanical or electronic device requiring precise position and direction control, and may be a stage light, an air conditioning device, a camera, etc.
[0040] The infrastructure device (400) is attached to a rotatable mechanism and may be a structure capable of changing the direction of the infrastructure based on fourth positional relationship information.
[0041] Figure 2 is a diagram illustrating the detailed configuration of the server shown in Figure 1.
[0042] The server (100) may include a communication module (110), a processor (120), and memory (130).
[0043] The communication module (110) can transmit a control signal to control the infrastructure device.
[0044] The communication module (110) may include a device comprising hardware and software necessary to transmit and receive signals, such as control signals or data signals, through a wired or wireless connection with another network device.
[0045] The processor (120) may include various types of devices for controlling and processing data. The processor (120) may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed by code or instructions included in a program.
[0046] In one example, the processor (120) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the invention is not limited thereto.
[0047] The processor (120) performs operations according to the code stored in memory (130).
[0048] The memory (130) can store at least one of the information and data input to the communication module (110), the information and data required for the function performed by the processor (120), and the data generated according to the execution of the processor (120).
[0049] Memory (130) should be interpreted as a general term for non-volatile storage devices that retain stored information even when power is not supplied, and volatile storage devices that require power to retain stored information. In addition to volatile storage devices that require power to retain stored information, memory (130) may include cloud storage, SSD, magnetic storage media, or flash storage media, but the scope of the present invention is not limited thereto.
[0050] Memory (130) is electrically connected to the processor (120) and stores at least one code executed by the processor (120). Memory (130) stores code that causes the processor (120) to perform the following functions and procedures when executed through the processor (120).
[0051] In memory (130), code that causes calibration between the sensor and the marker placed in a specific space based on a marker detection algorithm corresponding to the type of marker may be stored.
[0052] Here, the sensor is positioned at a location where the marker enters the field of view, and the sensor's internal parameters, such as focus, principal point, and distortion coefficient, may be preset.
[0053] In addition, calibration can be performed based on homogeneous transformation.
[0054] The memory (130) may store code that causes internal parameters to be obtained by utilizing various camera internal calibration methods. Here, the sensor may be positioned vertically with the infrastructure device or together with the infrastructure device.
[0055] In the memory (130), a code is stored that causes the sensor to receive first positional relationship information between the marker and the sensor based on calibration between the marker placed in an arbitrary area of a specific space and the sensor that recognizes the marker. Here, the first positional relationship information defines the relative position and direction between the coordinate system of the first object and the coordinate system of the second object, and may be in the form of a homogeneous transformation matrix. For example, the first positional relationship information may define the relative position and direction between the coordinate system of the sensor and the coordinate system of the marker.
[0056] In the memory (130), code is stored that causes to derive a third position relationship information between the infrastructure device and the marker based on the first position relationship information and the second position relationship information between the infrastructure device and the sensor, which is placed in an area other than the area where the sensor and the marker are located within the specific space.
[0057] In the memory (130), when the infrastructure device receives a control input that causes it to perform an operation on a target area within the specific space, code is stored that causes it to derive a fourth position relationship information between the target area and the infrastructure device based on the coordinates of the target area and the third position relationship information.
[0058] Memory (130) may store code that causes to produce fourth position relationship information for controlling an infrastructure device according to a preset kinematic methodology based on a third coordinate change relationship. Here, the preset kinematic methodology may be Denavit-Hartenberg parameters.
[0059] The memory (130) may store code that generates a control signal for controlling an infrastructure device based on fourth position relationship information and transmits the control signal to the infrastructure device to cause the infrastructure device to be controlled. Here, the control signal may include at least one of a position, a direction, and an operating state.
[0060] In memory (130), code that causes to derive fourth position relationship information even without a sensor and marker after the position transformation relationship estimation step can be stored.
[0061] For example, since all control objectives are defined by first positional relationship information based on a marker, it is possible to convert to absolute coordinates through already stored relationships even without the marker itself. Accordingly, it is composed solely of stored conversion relationships and input work objectives, and fourth positional relationship information can be derived even without sensors and markers.
[0062] FIG. 3 is a diagram illustrating an example for calculating the coordinate transformation relationship between a sensor and a marker according to an embodiment of the present invention.
[0063] Referring to FIG. 3, the coordinate transformation relationship between the sensor (200) and the marker (300) can be derived through external calibration based on the marker (300).
[0064] The infrastructure device integrated control device can derive a transformation relationship parameter between two coordinate systems from each coordinate system between the marker (300) and the sensor (200) that recognizes the marker (300).
[0065] For example, an infrastructure device integrated control device can derive relative position (x, y, z) and attitude (roll, pitch, yaw).
[0066] The marker (300) may have a specific pattern. The infrastructure device integrated control device can find an object using the specific pattern of the marker (300) and determine the size of the marker (300).
[0067] The infrastructure device integrated control device can find the corner point of the extracted marker (300) and use it to estimate the relative positional attitude value of the marker (300) and the sensor.
[0068] The infrastructure device integrated control device can determine the position of the marker (300) based on the estimated relative position attitude value.
[0069] The infrastructure device integrated control device can derive position attitude fourth position relationship information for each infrastructure based on the position of the marker (300) through multiple infrastructures and sensors (200) with their relative positions determined.
[0070] FIG. 4 is a diagram illustrating an example for deriving fourth position relationship information of an infrastructure device according to an embodiment of the present invention.
[0071] Referring to FIG. 4, the integrated control device of the infrastructure device (400) can derive infrastructure fourth position relationship information centered on a reference point through calibration.
[0072] The integrated control device of the infrastructure device (400) can derive a control output value based on the coordinate system of the infrastructure device (400) by utilizing homogeneous transformation characteristics.
[0073] The integrated control device of the infrastructure device (400) can derive first position relationship information of the infrastructure device (400) by performing a preliminary estimation process to estimate the position and attitude transformation relationship between the marker (300) and the infrastructure device (400) and a control process to control the infrastructure based on the estimated relationship.
[0074] The integrated control device of the infrastructure device (400) can perform external calibration by selecting a Marker detection method according to the type of marker (300).
[0075] The integrated control device of the infrastructure device (400) can calculate the first position relationship information (T1) of the sensor (200) reference marker (300) estimated as a result of calibration, and define the third position relationship information (T3) between the infrastructure device (400) and the marker (300) by calculating the first position relationship information of the sensor (200) reference marker (300) and the position relationship information (T2) pre-set for each infrastructure.
[0076] The integrated control device of the infrastructure device (400) can control the direction of the infrastructure using the location of the marker (300) as a reference point based on the third position relationship information (T3).
[0077] When the integrated control device of the infrastructure device (400) inputs the coordinate change relationship (Tc_1) to the control target using the coordinates of the marker (300) as a reference point, it can obtain the fourth coordinate change relationship (T4) between the control target and the infrastructure device (400) coordinate system based on the third position relationship information (T3).
[0078] The integrated control device of the infrastructure device (400) can derive fourth position relationship information by applying the fourth coordinate change relationship (T4) to the kinematic methodology.
[0079] FIG. 5 is a diagram illustrating an example of controlling an infrastructure device according to an embodiment of the present invention.
[0080] Referring to FIG. 5, the sensor (200) can perform calibration with a marker (300) placed in a specific space to produce first position relationship information.
[0081] The infrastructure device integrated control device (100) can receive first position relationship information from the sensor (200).
[0082] The infrastructure device integrated control device (100) can derive third position relationship information between the infrastructure device and the marker based on the first position relationship information and the second position relationship information between the infrastructure device and the sensor, which are placed in an area other than the area where the sensor and the marker are located within the specific space.
[0083] The infrastructure device integrated control device (100) derives fourth position relationship information of the marker (300) location-based infrastructure device (400) based on the coordinates of the target area and the third position relationship information, and can control the infrastructure device (400) based on the fourth position relationship information. Here, the target area may be an area corresponding to the user's control input received from the terminal.
[0084] The infrastructure device integrated control device (100) can generate a control signal to control the infrastructure device (400) based on the fourth position relationship information and transmit it to the infrastructure device (400).
[0085] FIG. 6 is a flowchart illustrating the sequence of an infrastructure device integrated control method according to another embodiment of the present invention.
[0086] The infrastructure device integrated control method described below can be performed by the infrastructure device integrated control device or server (100) described above with reference to FIGS. 1 to 5. Accordingly, the content of the embodiment of the present disclosure described above with reference to FIGS. 1 to 5 can be applied in the same way to the embodiment described below, and content that overlaps with the description above will be omitted. The steps described below do not necessarily have to be performed in order, the order of the steps can be set in various ways, and the steps may be performed almost simultaneously.
[0087] Referring to FIG. 6, the infrastructure device integrated control method includes a first position relationship information calculation step (S100) between a sensor and a marker, a third position relationship information calculation step (S200) between an infrastructure device and a marker, and a fourth position relationship calculation step (S300).
[0088] The step of calculating first positional relationship information between a sensor and a marker (S100) is a step of receiving first positional relationship information between the marker and the sensor from the sensor based on calibration between a marker placed in an arbitrary area of a specific space and a sensor that recognizes the marker.
[0089] The first position relationship information calculation step (S100) between the sensor and the marker may include a step of performing calibration between the sensor and the marker based on a marker detection algorithm corresponding to the type of marker.
[0090] The step of calculating third position relationship information between an infrastructure device and a marker (S200) is a step of deriving third position relationship information between an infrastructure device and a marker based on the first position relationship information and the second position relationship information set between an infrastructure device and a sensor placed in an area other than the area where the sensor and the marker are located within the specific space.
[0091] The fourth position relationship calculation step (S300) is a step of deriving fourth position relationship information between the target area and the infrastructure device based on the coordinates of the target area and the third position relationship information when the infrastructure device receives a control input that causes it to perform an operation on a target area within the specific space.
[0092] A person skilled in the art to which this disclosure pertains will understand that, based on the foregoing description, other specific forms can be easily modified without altering the technical spirit or essential features of this disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of this disclosure is defined by the claims set forth below, and all modifications or variations derived from the meaning and scope of the claims and equivalents thereof should be interpreted as being included within the scope of this disclosure. The scope of this application is defined by the claims set forth below rather than by the foregoing detailed description, and all modifications or variations derived from the meaning and scope of the claims and equivalents thereof should be interpreted as being included within the scope of this application.
[0093] The form for carrying out the invention is substantially the same as the best form for carrying out the invention mentioned above.
[0094] The present invention has industrial applicability as it can be utilized for the control of infrastructure devices.
Claims
1. An infrastructure device integrated control method performed by at least one processor, wherein a) receiving from the sensor first positional relationship information between the marker and the sensor according to calibration between the marker placed in an arbitrary area of a specific space and the sensor that recognizes the marker; b) a step of deriving third position relationship information between the infrastructure device and the marker based on the first position relationship information and the second position relationship information between the infrastructure device and the sensor, which are placed in an area other than the area where the sensor and the marker are located within the specific space; and c) A method for integrated control of an infrastructure device, comprising the step of deriving fourth positional relationship information between the target area and the infrastructure device based on the coordinates of the target area and the third positional relationship information when the infrastructure device receives a control input that causes the infrastructure device to perform an operation on a target area within the specific space.
2. In Paragraph 1, The above first to fourth positional relationship information includes information regarding the relative positions and directions of the first object and the second object, and An infrastructure device integrated control method in which the above-mentioned first to fourth position relationship information is in the form of a homogeneous transformation matrix.
3. In Paragraph 1, An infrastructure device integrated control method in which the above calibration is performed based on homogeneous transformation.
4. In Paragraph 1, Step a) above is, An infrastructure device integrated control method comprising the step of performing calibration between the sensor and the marker based on a marker detection algorithm corresponding to the type of the marker.
5. In Paragraph 1, The internal parameters of the above sensor include at least one of a focus, a principal point, and a distortion coefficient, and An infrastructure device integrated control method in which the above internal parameters are preset based on user settings.
6. Communication module; At least one processor; and It includes a memory electrically connected to the processor and storing at least one code executed in the processor, When the above memory is executed through the above processor, the processor, An infrastructure device integrated control device that receives from the sensor first positional relationship information between the marker and the sensor based on calibration between the marker placed in an arbitrary area within a specific space and the sensor recognizing the marker, derives third positional relationship information between the infrastructure device and the marker based on the first positional relationship information and a preset second positional relationship information between the infrastructure device placed in an area outside the region where the sensor and the marker are located within the specific space, and when receiving a control input that causes the infrastructure device to perform an operation on a target area within the specific space, stores a code that causes the infrastructure device to derive fourth positional relationship information between the target area and the infrastructure device based on the coordinates of the target area and the third positional relationship information.
7. In Paragraph 6, The above first to fourth positional relationship information includes information regarding the relative positions and directions of the first object and the second object, and An infrastructure device integrated control device in which the above-mentioned first to fourth positional relationship information is in the form of a homogeneous transformation matrix.
8. In Paragraph 6, An infrastructure device integrated control device in which the above calibration is performed based on homogeneous transformation.
9. In Paragraph 6, The above memory allows the processor, An infrastructure device integrated control device that stores code causing calibration between the sensor and the marker based on a marker detection algorithm corresponding to the type of the marker.
10. In Paragraph 6, The internal parameters of the above sensor include at least one of a focus, a principal point, and a distortion coefficient, and An infrastructure device integrated control device in which the above internal parameters are pre-configured based on user settings.