Method and system for evaluating a control command of a remotely controllable device based on a time reference

Abstract

The invention relates to a method (100) for evaluating a control command for a remotely controllable device (10) on the basis of a time reference (S1, S2), comprising the following steps: - outputting (106), by means of an output unit (14) associated with the remotely controllable device (10), a time reference (S1, S2) provided by means of a processing unit (12) into a detection region (E1) of a capturing unit (16) associated with the remotely controllable device (10); - detecting (108), by means of the capturing unit (16), the detection region (E1) having the time reference (S2) output by the output unit (14); - transmitting (110), by means of the capturing unit (16), a first signal (S3), which represents the detection region (E1) with the time reference (S2), to a control unit (18) arranged outside the remotely controllable device (10); - receiving (116), by means of the processing unit (12), a second signal (S4) which is provided by the control unit (16) on the basis of the first signal (S3) and represents the control command and the time reference (S1; S2); and - evaluating (118), by means of the processing unit (12), the control command on the basis of a comparison of the time reference (S1; S2) represented by the second signal (S4) with an expected value.

Description

[0001] R. 417142

[0002] - 1 -

[0003] Description

[0004] title

[0005] Method and system for evaluating a control command of a remotely controllable device based on a time reference

[0006] The invention relates to a method and a system for evaluating a control command based on a time reference. The invention also relates to a corresponding computer program and a computer-readable medium.

[0007] State of the art

[0008] US10482560 describes a method for securing a remote video transmission as follows:

[0009] A marker, digitally encoded in bits, is generated for an image. The encoded marker contains a camera identifier and a time variable. The encoded marker is inserted into the image. The encoded marker is then extracted from the secure image. The time variable is verified by comparing it with the camera's time reference and identifier to confirm the temporal recency of the secure image and its origin.

[0010] Disclosure of the invention

[0011] The disclosed invention proposes a method for evaluating a control command for a remotely controllable device based on a time reference, which comprises the following steps:

[0012] Output of a time reference provided by a computing unit into a detection range of a receiving unit assigned to the remotely controllable device by means of an output unit assigned to the remotely controllable device; R. 417142

[0013] - 2 -

[0014] Capturing the capture area, showing the time reference output by the output unit, using the recording unit;

[0015] Sending a first signal, representing the detection area with the time reference, to a control unit located outside the remotely controlled device by means of the recording unit;

[0016] Receiving a second signal, representing the control command and the time reference, provided by the control unit based on the first signal, by means of the processing unit; and

[0017] Evaluating the control command based on a comparison of the time reference represented by the second signal with an expected value using the processing unit.

[0018] Preferably, the process is carried out repeatedly in succession, in particular cyclically or periodically.

[0019] The disclosed invention further proposes a corresponding system for evaluating a control command for a remotely controllable device based on a (single) time reference according to claim 10.

[0020] The remotely controlled device can be, for example, a vehicle or a mobile or stationary machine, such as a manufacturing robot.

[0021] The computing unit preferably comprises a processor configured to perform the signal evaluation step. The computing unit may also include interfaces for communication and / or a storage medium. The communication interfaces can be used to send or receive signals, such as the time reference and / or the second signal. The computing unit may be configured, for example, as a processing unit, internal control unit, central processing unit, on-board control unit, or computer unit.

[0022] The time reference contains data that represents a time or a temporal reference and / or allows inferences to be made about it. Time here can be understood as local time or relative time, for example, after the processing unit has started. It is also conceivable that the time reference is a randomized sequence of numbers and / or characters, especially one generated by the system. 417142

[0023] - 3 -

[0024] includes which enables the computing unit to infer the time at which the time reference was provided to the output unit.

[0025] The output unit is configured to receive a time reference, preferably provided in digital form, via a communication interface and to output it in a manner that is detectable by a recording unit, in particular by displaying and / or projecting and / or outputting it as an audio signal. The output unit is exemplified as a monitor, projector, or loudspeaker.

[0026] The recording unit is configured to capture at least parts of its environment. The captured part of the environment can be two-dimensional or three-dimensional and can be designated as the capture area. Furthermore, the recording unit is configured to generate a first signal, which in particular represents the captured environment, preferably converting the captured environment into a digital signal. The recording unit is also configured to transmit the first signal to the control unit and / or other units via a communication interface. According to one embodiment, the recording unit is configured preferably to capture, detect, or record the visual environment, whereby the area visible to humans and / or the area invisible to humans can be captured. Alternatively or additionally, the recording unit can be configured to capture, record, or perceive its environment audibly.The recording unit is designed as a camera and / or microphone, for example.

[0027] The control unit is configured to receive signals, particularly the first signal, via a communication interface. The control unit can be configured to extract information from the received first signal and derive control commands from it. Alternatively, the control unit can be implemented as a control console with a user interface configured to display the first signal to a user and receive user input as a control command. The control unit is additionally configured to generate a second signal representing the control command and the corresponding time reference. Furthermore, the control unit is configured to send the second signal to the processing unit. The control unit is located outside of R. 417142

[0028] - 4 -

[0029] or located away from the remotely controlled device. For example, the control unit may be part of a cloud-based server application and / or a control center.

[0030] The method and system described above and below allow for the evaluation of the currency of the control unit's control command based on a single time reference. This proposes a particularly simple approach that eliminates the need for complex and error-prone synchronization of time references across multiple or all system units to ensure system functionality and safety. Furthermore, the proposed approach enables the use of standard components without extensive modifications, making the method and system applicable to a wide variety of applications.

[0031] In one embodiment, the output unit is designed as a visual output unit, in particular as a monitor and / or a QR code display, with a display within the detection range of the receiving unit. The output direction of the visual output unit is preferably oriented within the detection range and towards the receiving unit. This implementation of the method described above, with the integration of a visual output unit, allows this embodiment to be implemented with simple components and a large mechanical tolerance relative to the target position of the output unit.

[0032] In another embodiment, the time reference is output by projection into the detection area. Here, the time reference can be output by projection onto a part and / or an area within the detection area. It is also conceivable that the projection is directed at the recording unit and the time reference thus represents part of the detection area within an optical setup of the recording unit. For this purpose, a projection unit can be positioned or arranged near the recording unit. Depending on the design, the projection output by the projection unit can be onto a part and / or an area within the detection area of ​​the recording unit or directly into a part of the detection area. (R. 417142)

[0033] - 5 -

[0034] The optical input of the recording unit should be directed. This can facilitate the integration of the output unit through a variable installation position.

[0035] In another embodiment, additional information is integrated into the time reference. This additional information can include an identification or identifier of the remotely controlled device to facilitate the assignment of signals to the transmitting units. This can increase the information depth of the system. By appropriately modifying the time reference in the participating units, it is also possible to deduce the individual latencies of the first and / or second signal.

[0036] In another embodiment, the remotely controllable device is controlled by the processing unit using the control command provided by the control unit, if the comparison of the time reference with the expected value yields a difference below a first threshold. "Use" here can be understood to mean, in particular, the exact transmission of the control command from the control unit. The processing unit can also adapt, modify, or alter the control command, for example, if the sensors of the remotely controllable device or other associated units provide information that makes exact execution of the control command disadvantageous. The control command can, for example, be directed to an actuator and / or a sensor of the remotely controllable device, in particular, transmitted or sent. The control command can be configured to direct an actuator, for example.to control the steering of a remotely controlled device designed as a vehicle. This ensures that the control command is generated close enough in time to the situation recorded and transmitted by the recording unit.

[0037] In a further embodiment, the control command, in particular the transmission or forwarding of the control command, from the control unit to the remotely controllable device is prevented by the processing unit if the comparison of the time reference with the expected value results in a difference above the first threshold value. This has the advantage that no control command is executed by the remotely controllable device that is based on an insufficiently current situation in the detection range. In addition to preventing the R. 417142

[0038] - 6 -

[0039] In this embodiment, the computing unit can instruct an emergency stop or other safety measures in response to a control command, in particular the transmission or forwarding of a control command.

[0040] For both the first and second thresholds, a choice can be made based on preference and / or application, determining which option will be used if the difference is exactly equal to the value of the first or second threshold. Possible ranges for the first and / or second thresholds include 50 ps to 5 seconds, preferably 10 ms to 200 ms.

[0041] In another embodiment, the transmission rate of the recording unit and / or the control unit and / or the magnitude of the transmitted first and / or second signal are adjusted by the processing unit if comparing the time reference with the expected value results in a difference above a second threshold. For this purpose, the processing unit sends a command and / or information to, for example, the recording and / or control unit. The adjustment measures can be understood as preventive or necessary measures. It is advantageous if the second threshold is set lower than the first threshold. In this context, "preventive" means that the difference is below the first threshold but above the second. This ensures that the timeliness of the control command for the second signal is maintained.However, timeliness cannot be maintained or achieved solely by adjusting the transmission rate and / or amplitude of the first and / or second signal to the receiving and / or control unit using the processing unit. Adjusting the transmission rate preferably involves increasing the transmission rate. Adjusting the amplitude preferably involves decreasing the amplitude. By making at least one of the proposed adjustments, the difference between the time reference and the expected value used by the processing unit to evaluate the control command can be reduced.

[0042] In another embodiment, an information signal is provided by the processing unit. This information signal represents the evaluation of the received second signal. The representation of R. 417142

[0043] - 7 -

[0044] The evaluation can be the difference between the time reference and an expected value, expressed as a numerical value, and / or a color-coded system similar to a status indicator, allowing conclusions to be drawn about predefined ranges of the difference values. The control unit, whether operating automatically or with user input, can use this information signal as feedback on the system's performance. This feedback allows the user or the automated control unit to, if necessary, subject the sending of a further control command based on the first signal to a prior evaluation of the relevance of that command. If there is a large difference between the time reference and the expected value, the user and / or the automated control unit can suspend a control command of low relevance in order to achieve a smaller difference between the time reference and the expected value.

[0045] In another embodiment, the following steps are performed after the first signal has been sent:

[0046] Receiving the first signal by means of a further output unit located outside the remotely controlled device;

[0047] Output of the first signal via the further output unit;

[0048] Recording the time reference using a further recording unit located outside the remotely controlled device; and

[0049] Providing the second signal based on the recorded time reference using the control unit.

[0050] The additional output unit can, for example, be designed as a visual output unit. The first signal is displayed completely or partially, but at least the time reference represented by the first signal is displayed. The displayed time reference is received by another receiving unit, located outside the remotely controllable device, which can, for example, be designed as a visual receiving unit. The received time reference is made available by the additional receiving unit for the second signal to be sent by the control unit. Accordingly, the system can be designed according to this embodiment and additionally include the additional output unit and the additional receiving unit. This embodiment can relieve the control unit of some of its workload. In addition, this design allows the safety-relevant additional output unit to be controlled by another system, the additional R. 417142

[0051] - 8 -

[0052] The receiving unit is monitored. Therefore, the subsequent output unit does not need to comply with functional safety requirements.

[0053] Furthermore, it can be advantageous to include a computer-readable medium, preferably in non-volatile form, or alternatively a communication interface to such a computer-readable medium in the computing unit of the remotely controlled device. This computer-readable medium can contain a computer program with program code for carrying out the method described above or one of the possible variants or combinations thereof.

[0054] The invention will be explained in more detail below with reference to several drawings.

[0055] This shows

[0056] Fig. 1 shows a schematic representation of a remotely controllable device and a flowchart of a method for evaluating a control command for the device according to a first embodiment; and

[0057] Fig. 2 shows a schematic representation of a remotely controlled device and a flowchart of a method for evaluating a control command for the device according to a second embodiment.

[0058] Figure 1 shows a schematic representation of a remotely controllable device 10, a remote control device 20, and a method 100 for evaluating the control command for the remotely controllable device 10. The system comprises a computing unit 12, an output unit 14, and a receiving unit 16.

[0059] The processing unit 12 creates a preferably digital time reference S1 and sends it to the output unit 14 in step 102. The output unit 14 receives the time reference S1 in step 104. The output unit 14 outputs the time reference S2 into a detection area of ​​the recording unit 16 in step 106. (R. 417142)

[0060] - 9 -

[0061] Time reference S2 may be identical in content to or based on time reference S1. If time reference S2 is based on time reference S1, this could mean, for example, that the output unit performs a further conversion of the signal. In step 108, the acquisition unit 16 captures the detection area E1 with the output time reference S2. In step 110, the acquisition unit 16 sends a first signal S3, representing the detection area E1 and the output time reference S2, to a control unit 18 located in or on the remote control device 20. The control unit 18 generates a control command based on the captured detection area E1 and, in step 114, sends this as a second signal S4, which also includes the time reference, to the processing unit 12. The processing unit 12 receives the second signal S4 in step 116.Subsequently, the computing unit 12 compares the time reference S1 contained in the second signal S4 with an expected value and, in step 118, evaluates the control commands of the control unit 18 based on this comparison.

[0062] According to this first embodiment, the remotely controllable device 10 is, for example, designed as a remotely controlled, possibly semi-autonomous vehicle or a stationary or mobile robot. The recording unit is, for example, designed as a camera and the time reference as a QR code and / or numerical value.

[0063] Figure 2 shows a schematic representation of a remotely controlled device and a flowchart of a method for evaluating a control command for the device according to a second embodiment.

[0064] Figure 2 adopts the nomenclature from Figure 1 and addresses the differences in the representation. Compared to Figure 1, the remote-controlled device 10 additionally features an actuator unit 24. The remote control 20, in addition to the control unit 18, features a further output unit 20 and a further receiving unit 22. The function of the units additionally included in Figure 2 and their interaction with existing units are described below.

[0065] In step 120, the computing unit 12 sends the received control command S5, which is included in the second signal S4, to the actuator unit 24 of the remotely controllable device 10. In step 122, the actuator unit 24 receives the control command S5. In step 124, the computing unit 12 sends a command and / or information. R. 417142

[0066] -less to, for example, the recording unit 16 and / or the control unit 18. This command and / or this information S6 is designed to influence, for example, the transmission rate and / or the file size of the first and / or second signal S3, S4. Thus, by increasing the transmission rate and / or decreasing the file size, a reduction in the difference between the time reference and the expected value can be achieved.

[0067] In step 110, in addition to the first signal S3, an identical signal S7 can be sent by the receiving unit 16 to the further output unit 20. In step 126, the further output unit 20 outputs at least the time reference S2, which is encompassed by signal S7, as signal S8. In step 128, the further receiving unit 22 captures signal S8 or at least the time reference S2 encompassed by it. In step 130, the captured time reference S2 or the captured signal S8 is made available to the control unit 18 as signal S9. The control unit 18 can integrate the time reference S2, encompassed by signals S7, S8, and S9, into the second signal S4.

Claims

R. 417142 - 11 - Claims 1. Method (100) for evaluating a control command for a remotely controlled device (10) based on a time reference (S1, S2), comprising the following steps: Output (106) of a time reference (S1, S2) provided by means of a computing unit (12) into a detection area (E1) of a recording unit (16) assigned to the remote controllable device (10) by means of an output unit (14) assigned to the remote controllable device (10); Capturing (108) the capture area (E1) showing the time reference (S2) output by the output unit (14) using the capture unit (16); Sending (110) a first signal (S3) representing the detection area (E1) with the time reference (S2) to a control unit (18) located outside the remotely controllable device (10) by means of the recording unit (16); Receiving (116) a second signal (S4) provided by the control unit (16) on the basis of the first signal (S3), which represents the control command and the time reference (S1;S2), by means of the processing unit (12); and Evaluating (118) the control command based on a comparison of the time reference (S1 ;S2) represented by the second signal (S4) with an expected value using the computing unit (12).

2. Method (100) according to claim 1, characterized in that the output unit (14) is designed as a visual output unit with a display in the detection area (E1) of the receiving unit (16).

3. Method (100) according to one of the preceding claims, characterized in that the time reference (S1; S2) is output by a projection into the detection area (E1). R. 417142 - 12 - 4. Method (100) according to one of the preceding claims, characterized in that further information is integrated into the output time reference (S1 ;S2).

5. Method (100) according to one of the preceding claims, characterized in that after the step of evaluating (118) the signal, the following step is carried out: Control of the remotely controlled device (10) using the control command (S4) provided by the control unit (18) by means of the computing unit (12) when the comparison of the time reference (S1) with the expected value results in a difference below a first threshold value.

6. Method (100) according to claim 5, characterized in that after the step of evaluating (118) the signal, the following step is carried out: Suppressing the control command from the control unit (18) to the remotely controllable device (10) by means of the computing unit (12) if the comparison of the time reference (S1) with the expected value results in a difference above the first threshold.

7. Method (100) according to claim 6, characterized in that after the step of evaluating (118) the signal, the following step is carried out: Adjusting a transmission rate of the recording unit (16) and / or the control unit (18) and / or a size of the transmitted first (S3) and / or second signal (S4) by means of the computing unit (12) when comparing the time reference (S1) with the expected value results in a difference above a second threshold.

8. Method (100) according to one of the preceding claims, characterized in that after the step of evaluating (118) the signal, the following step is carried out: R. 417142 - 13 - Providing an information signal to represent the evaluation of the signal (S4) using the computing unit (12).

9. Method (100) according to one of the preceding claims, characterized in that after the step of sending (110) the first signal, the following steps are carried out: Receiving the first signal (S3) by means of a further output unit (20) arranged outside the remotely controllable device (10); Output (126) of the first signal (S3) by means of the further output unit (20); Recording (128) the time reference (S1) by means of a further recording unit (22) arranged outside the remotely controlled device (10); and Providing the second signal (S4) based on the recorded time reference using the control unit.

10. System for evaluating a control command for a remotely controlled device (10) based on a time reference (S1, S2) comprising an output unit (14) which is configured to output a time reference (S1) into a detection area (E1) of a recording unit (16) in the form of a detectable time reference (S2), a receiving unit (16) which is configured to have the detection area (E1) to detect (108) the time reference (S2) output by the output unit (14) and to send (110) a first signal (S3) representing the detection area (E1) with the time reference (S2) to a control unit (18) arranged outside the remotely controllable device (10). a computing unit (12) configured to provide the time reference (S1) for the output unit (14) (102), to receive a second signal (S4) provided by the control unit (18), which represents the control command and the time reference (S1; S2) (118), and to evaluate the control command based on a comparison of the time reference with an expected value (118). R. 417142 - 14 - 11. System according to claim 10, characterized by: a further output unit (20) which is configured to receive and output (126) the first signal (S3) sent, and a further receiving unit (22) which is designed to receive (128) the time reference output by means of the further output unit (20) and to make available (130) the second signal (S4) to the control unit.

12. Computer program comprising instructions that cause the system of claims 10 or 11 to perform the method steps according to any one of claims 1 to 9.

13. Computer-readable medium on which the computer program according to claim 12 is stored.

Images