Method for interacting with a unit situated in a real environment, mobile device, and system
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2026-02-05
- Publication Date
- 2026-08-13
Smart Images
Figure EP2026053004_13082026_PF_FP_ABST
Abstract
Description
[0001] DLG250101PDE
[0002] 1
[0003] Description
[0004] Method for interacting with a unit, mobile device and system located in a real environment
[0005] The invention relates to a method and a mobile device for interacting with a unit arranged in a real environment, as well as a system.
[0006] The basic concept of Augmented Reality (AR) has existed for several decades and refers to the overlaying of real-time images of reality (e.g., camera images) with virtual information.
[0007] EP 3347878 B1 describes a method for overlaying a virtual image of a real-world scene. The virtual image is displayed using a mobile device. To position the virtual image correctly within the real-world scene, a QR code located in the real-world scene is used, for example. Advantageously, the virtual image can remain correctly positioned within the real-world scene even if the mobile device moves and the real-world scene changes accordingly.
[0008] The invention aims to create an improved method for interacting with a unit arranged in a real environment, an improved mobile device, and an improved system.
[0009] This problem is solved by a method for interacting with a unit arranged in a real environment, a mobile device, and a system according to the main claims. Advantageous embodiments and further developments of the invention are set forth in the dependent claims.
[0010] The described approach enables the realization of a complete system that allows sensory data from industrial and technical plants and devices to be visualized in real time via embedded sensors in an AR environment generated by the mobile device, thus providing a user with complex technical information. DLG250101PDE
[0011] 2
[0012] and to integrate real-time data directly into a real-world work environment. This allows the operator to make immediate decisions, change setpoints in a display on the mobile device, or control actuators, thus actively shaping the control of industrial and technical systems and equipment via the mobile device.
[0013] For example, the operator can stand in front of the system with a mobile device and view it as an AR representation. At least one unit, sensor and / or actuator, can be displayed in the correct position within the AR view. Furthermore, data read from the unit can be visually displayed, or an input field for data to be transmitted to the unit can be visually presented. Information on how to use the unit can also be visually displayed, enabling the operator to control or maintain even an unfamiliar system.
[0014] A method for interacting with a unit located in a real-world environment using a mobile device, wherein the unit comprises a sensor and / or an actuator and is wirelessly networkable, the method comprising the following steps:
[0015] Detecting a marker located in the real-world environment using an environment sensing device of the mobile device;
[0016] Determining marker data associated with the marker using a determination device of the mobile device, wherein the marker data includes visualization data for overlaying a visualization of the unit onto an image of the real environment displayed using a display device of the mobile device, positioning data for positioning the visualization in the image, and communication data for establishing a communication link between the mobile device and the unit;
[0017] Representing the image of the real environment and visualizing the unit using the display device, wherein the visualization is superimposed onto the image using the visualization data and the positioning data; andDL0250101PDE
[0018] 3
[0019] Establishing the communication link between the mobile device and the unit using a communication device of the mobile device, wherein the communication link is established using the communication data obtained from the marker.
[0020] The real-world environment could be, for example, a room in a building containing a technical system. This system could be, for instance, an industrial machine or piece of technical equipment. The system can be equipped with at least one unit, such as one or more sensors, and additionally or alternatively with one or more actuators that can be wirelessly networked. Such a sensor might be designed, for example, to detect the operating state of the system or the state of a medium processed by the system. Such an actuator might be designed, for example, to change the operating state of the system. Wireless networking can be achieved using a suitable communication standard, such as Bluetooth Low Energy.
[0021] The marker may have been specifically positioned adjacent to the at least one unit for the purpose of carrying out the approach described here, for example, on the technical equipment or in its immediate vicinity. The marker may be positioned so that it can be detected simultaneously with the at least one unit using the environmental sensing device of the mobile device. The mobile device may be a portable device that the operator can, for example, hold in their hand or wear on their head. The environmental sensing device may, for example, include a camera. For example, at least at the beginning of the execution of the procedure described here, the operator may position the mobile device so that the environmental sensing device can detect the marker. Marker detection may be carried out, for example, using suitable image analysis.
[0022] Depending on the embodiment, the marker data can be encoded in the marker, or the marker can contain a link, for example a short URL, that allows data retrieval of the marker data, for example from a storage device of the mobile device or from a storage device located externally to the mobile device. Thus, depending on the embodiment, determining the marker data can involve decoding the DLG250101PDE
[0023] 4
[0024] Marker data can be derived from an image of the marker or retrieved or read via an interface to the storage device. The marker data can thus be associated with the marker by being encoded in the marker itself or by being stored with the marker, for example, in the mobile device. The visualization data can include, for example, a realistic or a schematic visualization of the unit. The visualization can be designed so that it is clear to the operator what type of unit is being visualized. The positioning data can, for example, define the spatial location of the unit relative to the spatial location of the marker. The marker data can also simply contain connection data to an external data collection and selection information for choosing the specific scene.The mobile device can establish a connection to the external data collection, select the correct scene from this data collection, and download it to the mobile device.
[0025] Using positioning data, the visualization of the unit can be integrated into the representation of the real environment in such a way that, from the operator's perspective, the visualization is located in the representation at the position that corresponds to the unit's actual position in the real environment. Thus, the visualization can be integrated into the representation of the real environment with perspective and spatial accuracy. The process of displaying the representation of the real environment and the visualization embedded within it can be continuously repeated. Even when the mobile device is moving, the visualization remains perspectively and spatially accurate, as long as the unit is within the detection range of the environmental sensing device.Since the positioning data can define a real position of the unit, the visualization can be embedded in the image even if the unit itself is hidden, for example, behind a wall or a machine cover.
[0026] For displaying an image of the real environment, rendering methods known from augmented reality (AR) can be used. For the positionally accurate display of the unit's visualization, even when the mobile device is moved, methods known from EP 3347878 B1, for example, can be used. This enables the visualization to track its position accurately even when the mobile device is panned and the resulting change in the representation of the real environment. Thus, the step of displaying DLG250101PDE
[0027] 5
[0028] can be executed permanently, or repeatedly, for example, in response to movement of the mobile device and / or the receipt of data via the communication link.
[0029] The communication data can include, for example, the address of the unit, which the mobile device uses to address the unit and establish the communication connection. The communication connection can be established, for example, using the mobile device's antenna. The communication connection can enable unidirectional or bidirectional data transmission between the mobile device and the unit. This allows, for example, sensor data acquired by a sensor to be transmitted to the mobile device. Furthermore, operator input via the mobile device can be transmitted to the unit. The establishment step can include maintaining the communication connection. Thus, data can be continuously exchanged between the mobile device and the unit via the communication link.
[0030] For example, data from the unit can be queried and updated multiple times per second. Data exchange between the mobile device and the unit can occur at a frequency of at least 5 Hz. For simply displaying data received from the unit, 1 Hz (or even 0.2 Hz if necessary) may suffice, depending on the data being queried. There is also likely metadata or control information that is retrieved or sent once at the beginning of a session.
[0031] The procedure can include a step of acquiring environmental data depicting the real environment using the mobile device's environmental sensing device. In the detection step, the marker can be identified using the environmental data. In the display step, the image of the real environment can be displayed using the environmental data. The environmental data can, for example, represent a pictorial image of the real environment. The marker can be automatically detected and evaluated by appropriate analysis of the environmental data, for example, by decoding it. The environmental data can be processed to display the image, for example, on a display of the display device or to project it into the operator's eye using a projector of the display device. The steps of the acquisition DL0250101PDE
[0032] 6
[0033] The display and rendering processes can be repeated continuously to provide the operator with a constantly updated image. The visualization can be superimposed onto the current image at any given time.
[0034] The steps of capturing environmental data and displaying that image can be continuously repeated at a set frequency. This allows for the continuous acquisition of current environmental data and the continuous display of an updated image of the real environment and its visualization. The visualization can be continuously overlaid onto the current image using both the visualization data and the positioning data. This ensures that the visualization remains perspectively and spatially accurate even when the mobile device is in motion.
[0035] The process can include a step of defining an anchor point using environmental data acquired by the environmental sensing device. The anchor point can be assigned anchor-based positioning data. In the rendering step, the visualization can be overlaid onto the image using this anchor-based positioning data. In this way, the anchor point can be used in addition to or as an alternative to the marker to display the visualization in the correct location. Advantageously, new anchor points can be continuously defined, allowing the operator to move relatively freely within the real-world environment and still have the visualization displayed, even if the marker is no longer detectable by the environmental sensing device.
[0036] The definition step can therefore be repeated to consecutively define and link further anchor points. In this way, a chain of anchor points can be used as a navigational aid for the mobile device.
[0037] For example, the visualization can include a two- or three-dimensional model of the unit. The model can be chosen, for instance, so that even an operator unfamiliar with the equipment supporting the unit can recognize what type of unit it is.
[0038] The visualization can include a readout interface for displaying unit values received via the communication link and / or a DLQ250101PDE.
[0039] 7
[0040] The user interface includes a user interface for inputting data that can be transmitted to the unit via the communication link. For example, the visualization can include a pointer display or a digital display of one or more values received by the unit. This allows the operator to read the unit even if the unit itself does not have a reading device. For example, the visualization can include a switch, a slider, or a keypad that allows the operator to input data, which is transmitted to the unit, by appropriate action, such as a finger, a gesture, or eye movement. Depending on the embodiment, the display device can be touch-sensitive, operated via a mouse or an additional Bluetooth control device, or by evaluating a gesture or eye movement of the operator.In this way, the operator can be provided with a complete human-machine interface for operating the unit via the visualization.
[0041] The values can be received via the communication link and displayed using the visualization's readout interface during the display step.
[0042] Accordingly, the process can include a step of reading the data via the visualization's user interface, whereby the data can be transmitted via the communication link. This enables easy interaction between the operator and the unit.
[0043] A marker can be understood as an artificial marker placed in the scene, for example, a geometric marker in the form of a code or pictogram. The marker can be an artificial marker in the form of a one-dimensional or two-dimensional code. For example, the marker can be a matrix with light and dark areas. The marker can represent optoelectronically readable text. Data can be represented within the marker in the form of a symbol.
[0044] In the recognition step, the marker can be identified in the form of a QR code. For recognizing a QR code, or any future developments thereof, it is advantageous to use proven and robust recognition methods. Alternatively or additionally to a QR code, the marker can be recognized, for example, as an image marker, a matrix code, or a three-dimensional reference object. DL0250101PDE
[0045] 8
[0046] In the determination step, the marker data can include a setpoint for the unit. This setpoint can be transmitted to the unit via the communication link to adjust the unit to the setpoint. For example, the unit can be configured to compare the setpoint with an actual value and, depending on the result of the comparison, provide a control signal for controlling, for example, a plant that can be controlled using the unit.
[0047] In the determination step, the marker data can be decoded from an image of the marker using the determination device. This allows the process to be executed very autonomously. Additionally or alternatively, the marker data can be read from a storage device using the determination device and an address assigned to the marker or decoded from the marker image. The address could, for example, be a short URL. The storage device can be integrated into the mobile device or located externally. In this case, the determination step can first include an instruction to establish a connection and read the storage device from the marker image. The marker data can then be read from the storage device. Storing the marker data in the storage device makes executing the marker relatively simple.Thus, the process can be implemented in both an offline and an online version.
[0048] In the determination step, the marker data can include additional visualization data for overlaying another visualization of another unit onto the representation of the real environment displayed using the mobile device's display device, additional positioning data for positioning the additional visualization within the representation, and additional communication data for establishing another communication link between the mobile device and the additional unit. In the display step, the additional visualization of the additional unit can be displayed using the display device, with the additional visualization being overlaid onto the representation using the additional visualization data and the additional positioning data.During the setup phase, the communication link between the mobile device and the other unit can be established using the communication device and the additional communication data. Advantageously, this allows two or more units to be visualized using a single marker. The DLG250101PDE can be used for this purpose.
[0049] 9
[0050] The procedure can be extended accordingly so that additional data can be included in the marker data for each unit to be visualized.
[0051] According to one embodiment, in the determining step, the marker data can include further visualization data for displaying additional visualizations of further units in the image of the real environment shown using the mobile device's display device, further positioning data for positioning further visualizations in the image, and further communication data for establishing further communication links between the mobile device and the further units. In the displaying step, the further visualizations of the further units can be displayed using the display device, with the further visualizations being displayed in the image using the additional visualization data and the additional positioning data.During the setup phase, further communication links between the mobile device and the other units can be established using the communication equipment and the additional communication data.
[0052] When multiple communication connections are established with several different units, these can be established in parallel or sequentially, for example, repeatedly at intervals of a few seconds, milliseconds, or microseconds. A corresponding repetition frequency, for example, for repeated communication with the unit and / or for repeating a query sequence to several units in succession, can be between 0.1 Hz and 10 Hz. Thus, the communication connection and at least one other communication connection can be established sequentially, for example, with a repetition frequency between 0.1 Hz and 10 Hz.
[0053] The communication link can be established during the setup phase using Bluetooth Low Energy or a future successor to this standard. This offers a wide range of applications for the approach described here, as Bluetooth Low Energy is a common standard for wirelessly networked sensors and actuators. DLG250101PDE
[0054] 10
[0055] The mobile device can be a smartphone, a tablet, stereoscopic mixed reality glasses (wearable), or another head-mounted display for direct or indirect projection into the operator's eye without an external display. Therefore, the approach described here can be used on various devices that include a suitable environmental sensing device, a suitable display device, a suitable targeting device, and a suitable communication device. These devices can be implemented as separate units or, for example, at least partially as part of the mobile device's control unit.
[0056] For example, the steps of the procedure can be controlled or implemented, at least partially, using a microcontroller of the mobile device.
[0057] If the unit is configured as a sensor, the sensor can be designed to detect a physical and / or chemical quantity. For example, the sensor can be a temperature sensor, a flow sensor, a speed sensor, a voltmeter, or a spectrometer. If the unit is configured as an actuator, the actuator can be designed to switch an electrical circuit or a mechanical connection, for example, to operate a valve or drive an electric motor.
[0058] The approach presented here further creates a device designed to perform, control, and implement the steps of a variant of the method presented here in appropriate facilities. This embodiment of the invention in the form of a device also allows the problem underlying the invention to be solved quickly and efficiently.
[0059] The device can be configured to read input signals and, using these input signals, determine and provide output signals. An input signal can, for example, be a sensor signal readable via an input interface of the device. An output signal can be a control signal or a data signal that can be provided at an output interface of the device. The device can be configured to determine the output signals using a processing instruction implemented in hardware or software. For example, the device can include a logic circuit, an integrated circuit, or a software module and, for instance, be implemented as or comprised of a discrete component. DL0250101PDE
[0060] 11
[0061] A corresponding mobile device for interacting with a unit located in a real environment, wherein the unit comprises a sensor and / or an actuator and is wirelessly networkable, has suitable means for implementing the steps of an embodiment of the said method.
[0062] For example, the mobile device may have an environment sensing device configured to detect a marker located in the real-world environment. Furthermore, the mobile device may have a determination device configured to determine marker data associated with the marker, wherein the marker data includes visualization data for overlaying a visualization of the unit onto an image of the real-world environment displayed using a display device of the mobile device, positioning data for positioning the visualization within the image, and communication data for establishing a communication link between the mobile device and the unit.Furthermore, the mobile device can have a display unit configured to show an image of the real environment as well as a visualization of the unit, with the visualization being overlaid onto the image using visualization data and positioning data. The mobile device can also have a communication unit configured to establish a communication link between the mobile device and the unit, with the communication link being established using communication data. Optionally, the communication unit is configured to maintain the communication link continuously or intermittently.
[0063] A corresponding system has the following characteristics:
[0064] One embodiment of said mobile device;
[0065] the unit, which is positioned in the real environment and includes the sensor and / or the actuator and is wirelessly networkable; and
[0066] the marker located in the real environment to which the marker data is assigned. DL0250101PDE
[0067] 12
[0068] The system can therefore be expanded to include multiple units. These units can be identical or different in shape. Thus, the system can, for example, comprise two, three, four, or more units.
[0069] The system can also include a piece of equipment, such as a machine. In this case, at least one unit and optionally the marker can be located on the machine. The unit can be positioned on the equipment in a way that is visible to a person in the vicinity or concealed from view.
[0070] It is also advantageous to have a computer program product with program code that can be stored on a machine-readable medium such as semiconductor memory, hard disk memory or optical memory and is used to carry out the method according to one of the embodiments described above, if the program product is executed on a computer or device.
[0071] Exemplary embodiments of the invention are shown in the drawings and explained in more detail in the following description. It shows:
[0072] Fig. 1 shows an overview of an exemplary embodiment of a system;
[0073] Fig. 2 shows a schematic representation of an exemplary embodiment of a mobile device; and
[0074] Fig. 3 shows a flowchart of an embodiment of a method for interacting with a unit arranged in a real environment.
[0075] Fig. 1 shows an overview of an embodiment of a system 100 comprising a wirelessly networkable unit 102 and a marker 104, both arranged in a real-world environment. Optionally, the system 100 includes more than one unit 102, for example, a second unit 106 and a third unit 108. The number of units 102, 106, and 108 is chosen only as an example. Furthermore, the system includes a mobile device 110, which is, for example, carried or held by an operator. The mobile device 110 includes an environment sensing device that enables the detection of the marker 104. The mobile device 110 also includes a display device 112, here by way of example, a display that shows a DLG250101PDE
[0076] 13
[0077] Image 114 shows the real environment and a visualization 116 provides at least one unit 102. An image 117 of marker 104 and a second visualization 118 of the second unit 106 are also shown as examples.
[0078] For illustrative purposes only, units 102, 106, 108, and marker 104 are arranged on a technical system 120, for example, a machine, and a section of the technical system 120 is also represented using the display device 112 as part of the image 114 of the real environment. Optionally, the technical system 120 is a component of the system 100.
[0079] Units 102, 106, and 108 are, for example, sensors or actuators. For instance, at least one of units 102, 106, and 108 is configured as a sensor, and at least one other unit 102, 106, and 108 is configured as an actuator. According to one embodiment, units 102, 106, and 108 are configured as components of or coupled to the technical system 120. For example, unit 102 is configured as a sensor designed to detect a physical or chemical quantity, such as a temperature at a machine part of the technical system 120. For example, the second unit 104 is configured as an actuator designed to switch an electrical circuit of the technical system 120, for instance, to operate an electric drive or to actuate a solenoid valve.Units 102, 106, and 108 can be wirelessly networked and wirelessly coupled to the mobile device 110 via a suitable communication standard.
[0080] According to one embodiment, units 102, 106, and 108 each comprise their own housing. For example, the length and width of units 102, 106, and 108 are each less than 10 centimeters. Alternatively, at least one of units 102, 106, and 108 can be configured as an unhoused circuit board. According to one embodiment, units 102, 106, and 108 each have a power supply or energy harvesting device, such as a battery, a solar cell, a piezoelectric unit, or an electrical connection for power supply or for drawing energy from a switching current, and a transmit interface or a transmit / receive interface for wireless communication. According to one embodiment, units 102, 106, and 108 each lack a display for showing data and / or a mechanical input device for manually entering data. DL0250101PDE
[0081] 14
[0082] The approach described here allows the operator of the mobile device 110 to interact with at least one unit 102. For this purpose, the operator is, for example, in a position in front of the technical system 120 that allows him to align the mobile device 110 so that at least the marker 104 is detected.
[0083] To initiate the interaction with at least one unit 102, marker 104 is first detected. Predefined marker data, determined using the mobile device 110, is assigned to marker 104. For example, the marker data itself, or an instruction to read the marker data from a storage device, is machine-readable encoded in marker 104. The marker data includes suitable information that enables the mobile device 110 to correctly position the visualization 116 of unit 102 within the representation 114 of the real environment. Thus, the operator can see the visualization 116 at a position, in this case on the display, that corresponds, for example, to the actual position of unit 102.According to one embodiment, the marker data includes, on the one hand, visualization data, which for example includes instructions for the graphic design of the visualization 116, and on the other hand, positioning data, which enables the correct positioning of the visualization 116 in the image 114.
[0084] To communicate with unit 102, the marker data also includes communication data that enables the mobile device 110 to establish a communication link between itself and unit 102. This communication link allows, for example, sensor data to be transmitted from unit 102 to the mobile device 110. The sensor data can then be displayed as part of the visualization 116. Optionally, the communication link also allows data to be transmitted to unit 102. Such data could, for example, have been entered by the operator via an input interface included in the visualization 116.
[0085] According to one embodiment, the visualization 116 comprises at least one two- or three-dimensional model 122 of the unit 102. The model 122 represents, for example, a symbol or a pictorial representation of the unit 102. The model 122 enables the operator to recognize the presence and position of the unit 102 in the real environment, here on the technical system 120. DLG250101PDE
[0086] 15
[0087] According to one embodiment, the visualization 116 includes a reading interface 124, here for example in the form of a graphically represented pointer instrument. The reading interface 124 enables, for example, a graphical representation of values received via the communication link from unit 102.
[0088] Additionally or alternatively, the visualization 116, according to one embodiment, includes an operator interface 126, here for example in the form of a slider depicted graphically for selecting a setpoint. For example, the operator interface allows the operator to input data that is transmitted to the unit 102 via the communication link. For example, a setpoint entered accordingly or predetermined by the marker data for the unit 102 is transmitted to the unit 102 via the communication link in order to set the unit 102 to the setpoint.
[0089] Additionally or alternatively, the visualization 116 according to an exemplary embodiment includes a pictorial representation of a switch 128 for activating the unit 102.
[0090] According to one embodiment, the display device 112 is designed to be touch-sensitive, so that the operator can operate the operating interface 126, for example, with his finger.
[0091] According to an alternative embodiment, the display device 112 is configured to project the image 114 and the visualization 116 into the operator's eye. In this case, the mobile device 110 includes, for example, a sensor device that enables tracking of the operator's pupil or detection of a hand gesture, each of which can trigger operation of the user interface 126.
[0092] According to one embodiment, the visualization 116 includes a description 130 of the unit 102. The description 130 includes, for example, information about a type of unit and / or an operating manual for the unit 102.
[0093] According to one embodiment, model 122 is always shown when the portion of the real environment encompassed by image 114 includes unit 102. DL0250101PDE
[0094] 16
[0095] In contrast, according to one embodiment, other components of the visualization 116, such as the reading interface 124, are only shown if the model 122 is arranged approximately in the middle of the image 114, so that it can be assumed that the operator has focused his attention on the unit 102.
[0096] For example, visualization 118 of the second unit 106 therefore only includes one model 132 of the second unit.
[0097] The following example assumes that units 102, 106, and 108 are Bluetooth Low Energy devices.
[0098] The described approach enables augmented reality for intuitive interaction and immersive display of Bluetooth Low Energy device data, here using device data from units 102, 106, and 108 as examples.
[0099] This revolutionizes the way Bluetooth Low Energy (BLE) devices, and their sensor or actuator data, are made visible and usable for the operator. It leverages the fact that units 102, 106, and 108 can wirelessly send and transmit data over short distances to compatible devices such as smartphones or gateways, in this case, the mobile device 110, using BLE technology. The data is then displayed using the mobile device 110.
[0100] By combining Augmented Reality (AR), embedding an AR representation in reality, and the signals sent by units 102, 106, 108, for example in the form of BLE sensor modules, an intuitive and efficient interface is created that makes it possible to integrate, position, and use complex technical information and real-time data directly into the real working environment for further operating steps or decisions.
[0101] As an example, Fig. 1 shows the complete system 100, which enables the real-time visualization of sensor data from industrial and technical plants and equipment, for example, the technical plant 120, via units 102, 106, 108, for example in the form of embedded sensors, in an AR environment, here exemplified by the display device 112, and thus allows complex technical information and real-time data to be integrated directly into the operator's real working environment. DLQ250101PDE
[0102] 17
[0103] The operator can thus make immediate decisions, displayed on the portable display device, here in the form of the mobile device 110, which is, for example, a smartphone or tablet. It is possible to change setpoint values on units 102, 106, and 108, or to address units 102, 106, or 108 as actuators, thereby actively controlling the technical system 120.
[0104] According to one embodiment, at least one of the units 102, 106, 108 is designed as a sensor.
[0105] By scanning markers located and positioned in the real environment, such as image markers, QR codes, matrix codes or 3D reference objects, here exemplified by marker 104, which are attached to components or also mark certain areas of the system 120, or which are recognizable as markers due to the shape or marking of one of the units 102, 106, 108, e.g., implemented as sensors, or of a component such as the technical system 120 itself, data from units 102, 106, 108, e.g., BLE sensor data, are retrieved and embedded, for example, in the form of 3D models, overlays or other visual elements, such as visualizations 116, 118, into the user's field of view on a smartphone, tablet or other image-generating device, here exemplified by the mobile device 110.
[0106] The coded information in marker 104 plays a dual role:
[0107] For this to work, the positioning of the image displayed in the augmented reality, for example, model 122 of unit 102, must be embedded in reality, or, in this example, in image 114 of the real environment. Firstly, the position of the augmented reality visualization is embedded in image 114 of the real environment according to the positioning instructions taken from marker 104. Secondly, the positioning of the image displayed in the augmented reality, for example, model 122 of unit 102, must also be embedded in image 114 of the real environment, as shown here as an example. Thus, existing markers, image markers, QR codes, matrix codes, or 3D reference objects in reality, such as marker 104, serve as so-called world anchors.
[0108] When the viewer moves within the real environment, the procedure described in patent applications 102015 115394.7 and 102016 121 281.4 applies. DL0250101PDE
[0109] 18
[0110] Secondly, contact information for a required request to display the data, for example, from unit 102, as well as display information for the unit 102 to be connected or the units 102, 106, 108 to be connected to the mobile device 110, are stored. This coded meta-information represents a description of the communication method in order to be able to contact and retrieve data from various units 102, 106, 108, for example, different sensor types, in the most generic way possible.
[0111] According to one embodiment, a link is additionally or alternatively stored in marker 104, the data of which contains the access and the necessary retrieval procedures to this information, which is stored, for example, in other proprietary description forms.
[0112] This information, which is either encoded in marker 104 or can be read using a link encoded in marker 104, is also referred to as marker data.
[0113] According to one embodiment, control mechanisms are introduced into augmented reality. This allows, for example, at least one of the units 102, 106, or 108 to be sent a setpoint value. The difference between a displayed value, which is shown, for example, using the reading interface 124, and this setpoint value, which is entered or predetermined, for example, using the operating interface 126, is then evaluated by a control system of the technical system 120, and the system 120 is controlled accordingly.
[0114] According to one embodiment, the visualization 116 of the AR is designed to allow, for example, the contact and activation of unit 102 in the form of a BLE activator. Corresponding feedback then occurs via unit 102 or the corresponding units 102, 106, 108, whose display, here exemplified by the readout interface 124, changes accordingly and can thus be directly checked by the operator. Such contact and activation is enabled for unit 102, for example, via the operator interface 126 and the graphical representation of the power switch 128.
[0115] The following is an example representation of an AR scene using Fig. 1, in which installed and located BLE sensors, here units 102, 106, 108 are shown. DLG250101PDE
[0116] 19
[0117] Marker 104, shown here as a QR code, contains as marker data positioning information for an image displayed in augmented reality, in this case image 114 of the real environment. This positioning information is also referred to as positioning data.
[0118] Furthermore, the display device, here the mobile device 110, which displays the augmented reality, determines contact information, also referred to as communication data, or display information, also referred to as visualization data, from marker 104, for example, by reading it out. The unit 102, for example in the form of a BLE sensor, is contacted using the mobile device 110, also referred to as the display device, and the display information or data to be displayed is read out according to one embodiment. The display information or the data to be displayed can then be shown on the smartphone, tablet, or other technical image-generating display device, for example as part of the visualization 116.
[0119] The marker 104 or the secondary markers selected according to patent applications 102015 115394.7 and 102016 121 281.4 define the relative position of the device 110 to the marker 104 or the markers and thereby enables a predefined and fixed positioning of the visualization 116 of the unit 102 within the AR scene throughout the entire display of the AR scene.
[0120] If units 102, 106, and 108 are also activators, they can be interactively controlled from their representation in augmented reality.
[0121] The described approach can be used for learning scenarios with BLE sensors as units 102, 106, 108.
[0122] The following describes an implementation example that outlines a didactic learning environment in which measurement data from units 102, 106, and 108, in the form of BLE sensors, support a scientific experiment. The experiment takes place in the real world and is supplemented by a specially created AR scene. The AR scene provides instructions, additional information, and background knowledge about the experiments. As an example, such an AR scene is created using the 112DLQ250101PDE display device.
[0123] 20
[0124] depicted and includes, in addition to image 114 of the real environment, further information shown.
[0125] An innovative aspect of this solution is the ability to directly display and edit sensor values and configurations of units 102, 106, and 108 within the AR environment using the mobile device 110. This overlay enables an interactive and immersive data presentation. Ideally suited hardware includes mobile devices, such as the mobile device 110, for visualization, as well as units 102, 106, and 108 in the form of BLE sensors that capture physical measurements such as voltage, current, force, or pressure.
[0126] The test environment was set up as follows:
[0127] A standard smartphone, such as the Mobile Device 110, a Samsung Galaxy S21 or an Apple iPad 10th Gen., is used as a visualization device.
[0128] Commercially available BLE sensors in the form of a voltage sensor and a current sensor are selected as units 102, 106, and 108.
[0129] Marker 104 is a QR code containing information about the scene configuration (local and remote) and its real-world position. The scene configuration, in turn, includes detailed descriptions of scene content, such as text or interactive areas, in JSON format, as well as additional 3D models that may be necessary to describe the experimental setup. Access is via the URL encoded in the QR code, which allows data retrieval from a database. The QR code itself was positioned using the patented method described in the previously cited 3DQR patent applications.
[0130] The basis was a well-known experimental setup for "measuring voltage." This includes a voltage source and a DC circuit. The voltage sensor is connected in parallel to the DC circuit at the input / output of a light bulb, while the ammeter is integrated into the experimental setup in a series circuit.
[0131] A specially developed software for mobile devices (Android / iOS) generated an AR environment. After scanning marker 104 in the form of a QR code, DLG250101PDE
[0132] 21
[0133] Visualizations were downloaded via a scene configuration, interaction surfaces were defined which enable connecting or disconnecting as well as data polling, and the necessary configurations of the BLE sensors for communication were downloaded from the scene configuration.
[0134] The configuration of the BLE sensors contains information such as the MAC addresses of the BLE sensors, as well as necessary commands for connecting the BLE sensor and smartphone, and data polling in the form of defined byte strings that can be sent to the BLE sensor.
[0135] Example commands are Access device name: “OxCD” and Continuously Collect: “0xA2”.
[0136] The possible mobile devices, such as the mobile device 110, are then able to establish a connection to the BLE sensors upon user interaction on the display screen or the image-generating device, for example, the display unit 112. Commands were sent via byte sequences similar to those mentioned above, which access so-called Bluetooth GATT Services & Characteristics. Write characteristics are used to control the devices, while notify characteristics enable the reading of values.
[0137] Once a connection has been successfully established, metadata is read from the BLE sensor and data polling is started from the transmitted metadata.
[0138] The sensor data is then converted into data sets of physical quantities, in this case voltage and current. The visualization takes place at the positions predefined via the scene configuration on the imaging device, in this case the aforementioned smartphone; however, the scene can be displayed on any other imaging device that can generate the depicted scene.
[0139] The following is an application example from industry.
[0140] This example was based on a scenario using a standard BLE actuator or switch as unit 102.DLG250101PDE.
[0141] 22
[0142] The setup and procedure of the experiment largely mirror the previous example: The mobile device 110 scans marker 104 in the form of a QR code, loads a scene configuration, and assembles scene components at the positions specified by the downloaded scene rules. Here, too, configurations for connecting to unit 102 and the necessary communication commands are extracted. User interaction on the imaging device, mobile device 110, triggers a connection to, or a command is sent to, unit 102, i.e., the connected BLE actuator.
[0143] The experimental setup differs slightly.
[0144] For example, unit 102 is installed behind a drywall partition as technical equipment 120. Unit 102 is capable of switching a power connection to turn on a standard light bulb in an AC circuit. Marker 14, a QR code for scene entry, is located on the drywall partition. In the real-world environment, unit 102 is hidden behind the wall and not visible. Specially developed software for mobile devices (Android / iOS), in this case the mobile device, generates an AR environment, specifically in the form of image 114 of the real-world environment.After scanning marker 104 in the form of the QR code, the position of a virtual twin of unit 102, for example model 122, MAC addresses of unit 102, as well as necessary commands for the connection between unit 102 and the mobile device 110 and the data polling in the form of defined JSON objects which can be sent to unit 102 were downloaded.
[0145] The GATT characteristics (Generic Attribute Profile) are defined by the manufacturer as follows:
[0146] UUI D_RW: 5F6D4F53-5F52-5043-5F64-6174615F5F5F
[0147] UUID_READ_NOTIFY: 5F6D4F53-5F52-5043-5F72-785F63746C5F
[0148] UUID_W: 5F6D4F53-5F52-5043-5F74-785F63746C5F
[0149] With this information and the corresponding MAC address, mobile device 110 is able to establish a connection to unit 102 and also send commands to this unit 102. DLG250101PDE
[0150] 23
[0151] Interaction between mobile device 110 and unit 102 is possible after a connection has been established between mobile device 110 and unit 102, or units 102, 106, and 108. During the connection process, metadata from unit 102 is read. Sending JSON commands to the WriteCharacteristics (UUID_W) parameter, for example, toggles unit 102, essentially connecting or disconnecting the power supply. This allows the state of unit 102 to be changed. The following commands activate or deactivate unit 102:
[0152] {"id":"1641784XXX","src":"shelly-app","method":"switch.set","params":{"id":0,"on":false}} {"id":"1641784XXX","src":"shelly-app","method":"switch.set","params":{"id":0,"on":true}}
[0153] According to one embodiment, the commands are triggered from the mobile device 110 via a user button, for example the operating interface 126 and / or the pictorial representation of the power switch 128, the visualization 116 within the AR scene displayed using the display device 112.
[0154] Most BLE sensors are small, non-interactive interfaces, usually without a display. These devices are designed to provide and transmit sensor data over short to medium distances.
[0155] AR offers the additional dimension of being able to interactively and visually enhance these sensors on an imaging device such as a smartphone or laptop - very close to the location of the installed BLE sensor.
[0156] Sensor modules can thus be configured, controlled, and read via the AR scene, depending on their respective APIs. This technology is suitable for use on mobile devices such as smartphones and tablets, as well as on AR or MR-enabled HMDs (head-mounted displays). A basic requirement is the presence of a Bluetooth interface, which is certainly not only a general industrial standard these days. DLG250101PDE
[0157] 24
[0158] The described approach includes in particular the combination of QR codes, AR and BLE sensor data for visualization and interaction in industrial as well as educational environments, and the specific method of data transmission and display.
[0159] This offers an innovative partial solution to the challenges of Industry 4.0 and the Internet of Things. The merging of the physical and digital worlds heralds a new era of human-machine interaction.
[0160] Fig. 2 shows a schematic representation of an embodiment of a mobile device 110. The mobile device 110 is used by an operator, whose eye 200 is shown, to interact with a unit 102, for example, a wirelessly networkable sensor or a wirelessly networkable actuator. For example, the mobile device 110 and the unit 102 are elements as described with reference to Fig. 1.
[0161] The mobile device 110 comprises an environmental sensing device 230, a detection device 232, a display device 112, and a communication device 234. The environmental sensing device 230 includes, for example, a camera. The display device 112 is configured, for example, as a screen or as a projection device for projecting an image into the operator's eye 200. The communication device 234 includes, for example, a transceiver unit for wireless communication with the unit 102.
[0162] According to one embodiment, the devices 112, 230, 232, 234 are coupled in a signal-transmitting manner and can optionally use a common computing unit, for example a microprocessor of the mobile device 110.
[0163] The mobile device 110 is, for example, a smartphone, a tablet, a stereoscopic mixed reality glasses (wearable) or another head-mounted display.
[0164] During an initialization process, the environmental sensing device 230 is used to detect the marker 104. For example, the environmental sensing device 230 is configured to detect the marker 104 using image analysis of a captured image and to classify it, for example, as a QR code. DL0250101PDE
[0165] 25
[0166] Marker data is assigned to marker 104. This marker data is either encoded within marker 104 itself or stored for marker 104, for example, in a storage device 236. The marker data may also simply contain connection data to an external data collection and selection information for choosing the specific scene. The mobile device 110 can establish a connection to the external data collection, select the correct scene from this collection, and download it to the mobile device 110.
[0167] The identification device 232 is configured to determine the marker data associated with marker 104 in response to the detection of marker 104. For example, the identification device 232 is configured to decode the marker data from an image of marker 104 acquired using the environment detection device 230.
[0168] Additionally or alternatively, the identification device 232 is configured, for example, to decode a link from the image of the marker 104 or to select a corresponding link from a lookup table and to read the marker data from the storage device 236 using the link. The storage device 236 can be part of the mobile device 110 or arranged externally to the mobile device 110.
[0169] According to one embodiment, the marker data includes visualization data that defines a visualization 116 which can be displayed as a virtual representation of the unit 102 using the display device 112.
[0170] According to one embodiment, the marker data includes positioning data for positioning the visualization 116 in a representation of the real environment, displayed using the display device 112, in which the marker 104 and the unit 102 are located. The positioning data enables the visualization 116 to be integrated into the representation of the real environment such that the visualization 116 is located at a position of the unit 102 corresponding to that of the real environment. For example, the visualization 116 can be displayed such that the distance and orientation between the visualization 116 and a displayed representation of an object, such as the marker 104, are in a correct ratio to the actual distance and orientation between the unit 102 and the real object. DLG250101PDE
[0171] 26
[0172] According to one embodiment, the marker data includes communication data for establishing a communication link between the mobile device 110 and the unit 102. For example, the communication data includes an address for addressing the unit via the communication device 234 and a control command that can be sent via the communication link to control a function of the unit 102.
[0173] In an operational process following the initialization, the display device 112 is used to show a representation of the real environment and, using the visualization data, the visualization 116 of the unit. Using the positioning data, the display device is configured to overlay the visualization 116 onto the representation in the correct location. This representation allows the operator to determine the actual position of the unit 102 in the real environment, even if the unit 102 itself is not visible, for example, because it is located behind a wall.
[0174] According to one embodiment, the environmental sensing device 230 is configured to continuously acquire environmental data, for example at a predetermined clock frequency, which depicts the real environment. This continuously updated environmental data is used by the display device 112 to continuously display current images of the real environment. Furthermore, the display device 112 is configured to continuously overlay the visualization 116 onto the currently displayed image in the correct position.
[0175] According to one embodiment, the display unit 112 is designed to correctly position the visualization 116 within the current image even when the marker 104 is no longer detected by the environment detection unit 230 and is therefore not displayed in the current image. For this purpose, the determination unit 232 is designed, for example, to continuously define new anchor points, such as an anchor point 240, in the real environment using the environmental data acquired by the environment detection unit 230, and to provide these anchor points with anchor-based positioning data. This data can then be used by the display unit 230 to correctly position the visualization 116 relative to an image of the corresponding anchor point 240. Such anchor points can also be linked together, allowing the operator, for example, to completely orient themselves around a system equipped with the unit 102 or DLG250101PDE.
[0176] 27
[0177] It can also move around outside, while always maintaining a correct positioning of the virtualization 116. For example, a structure or object that is clearly identifiable using the environment detection device 230 is selected for the anchor point 240.
[0178] According to one embodiment, different anchor points 240 are consecutively defined and linked together. For example, linking two anchor points can define a relative position of the two anchor points to each other, and at least one of the two anchor points can be stored in the anchor-based positioning data.
[0179] The communication device 234 is used to establish and maintain the communication link between the mobile device 110 and the unit 102 using the communication data, according to one embodiment. For example, the communication device 234 is configured to establish and operate the communication link using Bluetooth Low Energy.
[0180] According to one embodiment, the communication device 234 is configured to establish at least one further communication connection to at least one further unit 106. The communication connections are maintained, for example, in parallel over time or sequentially, for example at a predetermined repetition frequency, for example between 0.1 Hz and 10 Hz.
[0181] To couple the multiple units 102, 106 with the mobile device 110, the marker data, according to one embodiment, includes corresponding visualization data, positioning data, and communication data for each of the units 102, 106. This enables the display unit 112 to overlay a corresponding visualization 116 onto the current representation of the real environment for each of the units 102, 106. For example, no visualization is currently displayed for the other unit 106, as it is not currently within the detection range of the environment detection unit 230. Furthermore, the communication unit 234 is enabled to establish a separate communication connection to each of the units 102, 106. DL0250101PDE
[0182] 28
[0183] According to one embodiment, the communication device 234 is configured to receive at least one value from the unit 102 via the communication link, and the display device 112 is configured to display the visualization 116 in such a way that the value received from the unit 102 is clearly displayed to the operator, for example using a readout interface as described with reference to Fig. 1.
[0184] According to one embodiment, the visualization 116 is configured to allow the operator to input data, for example, using an operator interface as described with reference to Fig. 1. In this case, the communication device 234 is configured to send a command representing the input to the unit 102. For example, the input can define an activation of the unit 102 or a value to be set on the unit 102.
[0185] The described approach allows the visualization of the unit to be positioned as a virtual optical object in a perspective-correct representation from a great distance and from a relatively unrestricted position of the mobile device. Advantageously, it is not necessary for the marker to be constantly within the field of view of the mobile device's environmental perception system.
[0186] According to one embodiment, the described approach uses measuring devices located in the mobile device—in addition to image acquisition—to measure changes in the relative orientation of the mobile device compared to its orientation at the time of initial marker detection. Additionally, data from a real object is used as an anchor point, also known as a "secondary marker," so that the actual marker no longer needs to be within the detection range of the environmental sensing device. Suitable measuring devices, also referred to as detection devices or measuring sensors, could include, for example, an accelerometer, a rotation sensor, a magnetometer, a GPS receiver, or a microphone.
[0187] The environmental sensing device can be limited to visible light (400-800nm), but can also additionally or exclusively detect other spectral ranges (e.g., additionally or exclusively IR or UV light). DL0250101PDE
[0188] 29
[0189] Fig. 3 shows a flowchart of an embodiment of a method for interacting with a unit located in a real environment using a mobile device, as described with reference to the preceding figures.
[0190] For example, the procedure can be carried out using the mobile device's features, as described in the preceding figures.
[0191] In step 301, at least one marker located in the real environment is detected.
[0192] In step 303, the marker data assigned to the marker is determined. The marker data includes visualization data, positioning data, and communication data.
[0193] In step 305, a representation of the real environment is displayed, with a visualization of the unit superimposed onto the representation using the visualization data and the positioning data. According to one embodiment, the visualization, for example, in particular a model of the unit, is superimposed onto the representation in a perspectively and spatially correct manner, corresponding to the real positioning of the unit.
[0194] In step 307, a communication link between the mobile device and the unit is established and optionally maintained using the communication data. Optionally, step 307 is executed continuously to maintain the communication link at a clocked rate. For example, the communication link is established repeatedly at a repetition frequency between 0.1 Hz and 10 Hz. This allows different communication links to different units to be established alternately and sequentially, and used for data transfer between the mobile device and the respective unit. Furthermore, data can be continuously exchanged between the mobile device and the unit.
[0195] Optionally, the procedure includes a step 309 in which environmental data representing the real environment is acquired. This environmental data is used in step 301 to detect the marker. In step 305, the image of the real environment is displayed using the environmental data. For example, steps 309, 305DL0250101PDE
[0196] 30
[0197] The process is executed continuously and repeatedly, so that even when the mobile device is in motion, current environmental data depicting the real environment is continuously captured and used to display a real-time representation of the environment. According to one embodiment, the visualization is always displayed perspectively and spatially correctly within the current image, provided the unit is located, even if partially obscured, within the current detection range of the mobile device.
[0198] According to one embodiment, the marker data includes a setpoint for the unit. In this case, the setpoint is transmitted to the unit via the communication link established in step 307. This allows the unit to be set to the setpoint.
[0199] According to one embodiment, the method comprises a step 311 in which data entered by an operator, for example via an operator interface of the visualization, is recorded and subsequently transmitted to the unit via the communication link established in step 307.
[0200] According to one embodiment, at least one value is received from the unit via the communication link established in step 307 and displayed to the operator in step 305 as part of the visualization, for example via a readout interface of the visualization.
[0201] According to one embodiment, the method includes step 313 of defining an anchor point using environmental data acquired by the environmental sensing device. The anchor point is provided with anchor-based positioning data, which can be used in a repeated rendering step to overlay the visualization onto the image. Step 313 can be executed repeatedly to continuously define new anchor points, which, for example, can be located increasingly farther from the original marker. As long as one of the anchor points is within the current sensing range of the mobile device, the visualization can be overlaid with the correct position. For example, an object or an edge in the real-world environment can be chosen as the anchor point.
Claims
DLG250101PDE 31 Claims 1. Method for interacting with a unit (102) arranged in a real environment using a mobile device (110), wherein the unit (102) comprises a sensor and / or an actuator and is wirelessly networkable, the method comprising the following steps: Detect (301) a marker (104) arranged in the real environment using an environment sensing device (230) of the mobile device (110); Determine (303) marker data associated with the marker (104) using a determination device (232) of the mobile device (110), wherein the marker data includes visualization data for overlaying a visualization (116) of the unit (102) onto an image (114) of the real environment displayed using a display device (112) of the mobile device (110), positioning data for positioning the visualization (116) in the image (114), and communication data for establishing a communication link between the mobile device (110) and the unit (102); Representing (305) the image (114) of the real environment and the visualization (116) of the unit (102) using the representation device (112), wherein the visualization (116) is superimposed onto the image (114) using the visualization data and the positioning data; and Establishing (307) the communication link between the mobile device (110) and the unit (102) using a communication device (234) of the mobile device (110), wherein the communication link is established using the communication data.
2. Method according to claim 1, comprising a step (309) of acquiring environmental data mapping the real environment using the DL0250101PDE 32 Environment sensing device (230) of the mobile device (110), wherein in step (301) of detection the markers (104) are detected using the environment data, and wherein in step (305) of rendering the image (114) of the real environment is rendered using the environment data.
3. Method according to claim 2, wherein the step (309) of acquiring environmental data depicting the real environment and the step (305) of displaying the image (114) of the real environment are continuously repeated at a clock frequency in order to continuously acquire current environmental data depicting the real environment and to continuously display a current image (114) of the real environment and the visualization (116), wherein the visualization (116) is continuously superimposed on the current image (114) using the visualization data and the positioning data.
4. Method according to claim 2 or 3, comprising a step (313) of defining an anchor point (240) using the environmental data acquired by the environment sensing device (230), wherein the anchor point (240) is provided with anchor-based positioning data, wherein in the step (305) of displaying the visualization (116) is superimposed onto the image (114) using the anchor-based positioning data.
5. Method according to claim 4, wherein step (313) is repeated to consecutively define and link further anchor points (240).
6. Method according to any of the preceding claims, wherein the visualization (116) comprises a two- or three-dimensional model (122) of the unit (102).
7. Method according to one of the preceding claims, wherein the visualization (116) comprises a readout interface (124) for displaying values received from the unit (102) via the communication link and / or an operator interface (126) for inputting data that can be sent to the unit (102) via the communication link.
8. Method according to claim 7, wherein the values are received via the communication link and displayed in step (305) of the display using the readout interface (124) of the visualization (116) and / or with a step (311) of the input DLG250101PDE 33 the data via the user interface (126) of the visualization (116), whereby the data are transmitted via the communication link.
9. Method according to one of the preceding claims, wherein in step (301) the marker (104) in the form of a QR code is recognized.
10. Method according to one of the preceding claims, wherein in the determining step (303) the marker data comprise a setpoint for the unit (102), wherein the setpoint is transmitted to the unit (102) via the communication link in order to set the unit (102) to the setpoint.
11. Method according to one of the preceding claims, wherein in step (303) of determining the marker data are decoded from an image of the marker (104) using the determining device (232) or are read from a storage device (236) using an address assigned to the marker (104) or decoded from the image of the marker (104).
12. A method according to any of the preceding claims, wherein in step (303) of determining, the marker data comprises further visualization data for displaying at least one further visualization (118) of at least one further unit (106) in the image (114) of the real environment displayed using the display device (112) of the mobile device (110), further positioning data for positioning the at least one further visualization (118) in the image (114), and further communication data for establishing at least one further communication connection between the mobile device (110) and the at least one further unit (106), wherein in step (305) of displaying, the at least one further visualization (118) of the at least one further unit (106) is displayed using the display device (112).wherein at least one further visualization (118) is superimposed onto the image (114) using the further visualization data and the further positioning data, and wherein in step (307) of the setup, at least one further communication link is established between the mobile device (110) and the at least one further unit (106) using the communication device (234) and the further communication data. DL0250101PDE 34 13. Method according to claim 12, wherein the communication link and the at least one further communication link are repeatedly established successively with a repetition frequency between 0.1 Hz and 10 Hz.
14. Method according to one of the preceding claims, wherein the communication link is established in step (307) of the setup using Bluetooth Low Energy.
15. Method according to any of the preceding claims, wherein the mobile device (110) is a smartphone, a tablet, a stereoscopic mixed reality glasses (wearable) or another head-mounted display.
16. Method according to any of the preceding claims, wherein in the case of a unit (102) designed as a sensor the sensor is designed to detect a physical or chemical quantity, and / or wherein in the case of a unit (102) designed as an actuator the actuator is designed to switch an electrical circuit or a mechanical connection.
17. Mobile device (110) for interacting with a unit (102) arranged in a real environment, wherein the unit (102) comprises a sensor and / or an actuator and is wirelessly networkable, wherein the mobile device (110) has means for implementing the steps of the method according to any of the preceding claims.
18. System with the following characteristics: a mobile device (110) according to claim 17; the unit (102) which is located in the real environment and includes the sensor and / or the actuator and is wirelessly networkable; and the marker (104) located in the real environment, to which the marker data are assigned.
19. System according to claim 18, comprising a technical installation (120), in particular a machine, wherein the unit (102) is arranged on the technical installation (120). DL0250101PDE 35 20. Computer program product with program code for carrying out the method according to any one of claims 1 to 16, when the computer program product is executed on a device.