Computer-implemented method for determining a marking of an object by means of a sensor

Abstract

The invention relates to a computer-implemented method (100) for determining a marking of an object by means of a sensor (200) which is designed to scan an environment, the method (100) comprising: generating (110) a raw dataset based on the scanned environment, wherein the raw dataset characterises a reflectivity of at least one object in the scanned environment; assigning (120) the object to a predetermined object group; providing (130) a marking database, which characterises an expected reflectivity of the object group; determining (140) a marking status of the assigned object based on a comparison of the reflectivity of the assigned object with the expected reflectivity of the assigned object group, wherein the marking status characterises the presence of a marking on the assigned object; outputting (150) a processed dataset, wherein the processed dataset characterises at least the assigned object and comprises the marking status of the assigned object.

Description

[0001] ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28

[0002] Computer-implemented method for determining a marker on an object using a sensor

[0003] The invention relates to a computer-implemented method for determining a marking of an object, a sensor, a device comprising the sensor, and a computer program product.

[0004] Intelligent sensors, also known as smart sensors, are sensors that combine the actual measurement process with complete signal conditioning and processing in a single unit. These complex sensors typically include a microprocessor or microcontroller and provide standardized interfaces for communication with higher-level systems, such as fieldbus systems, sensor networks, or IO-Link. Consequently, the output of such intelligent sensors does not consist of raw data, but rather of datasets derived from the raw data. These datasets contain less information than the raw data, as the raw data is already processed by the sensor to execute a specific instruction. A key advantage of intelligent sensors is that processing power can be offloaded to the sensors themselves.

[0005] In the area of ​​reflectors and markings, such datasets do not contain sufficient information regarding the reflectors or markings of objects scanned by the sensors. Therefore, the datasets cannot be used for further processing with respect to such reflectors and markings. It is not possible to deduce the presence of reflectors and markings. The raw data is not included with the datasets; this would also defeat the purpose of intelligent sensors.

[0006] The state of the art is known from DE 101 16 277 A1. This document describes a method and a device for detecting objects during the operation of a motor vehicle. A detection device is provided that can detect the size, reflectance, speed, acceleration, and / or direction of objects. The detection device can be a laser. A subsequent logic unit of the vehicle (ZF Friedrichshafen AG file 306016, Friedrichshafen, 2024-11-28) performs object identification based on a pattern matching procedure.

[0007] It is an object of the present invention to provide a computer-implemented method, a sensor, a device comprising the sensor, and a computer program product that improve upon or even eliminate one or more of the aforementioned disadvantages. In particular, it is an object of the present invention to utilize the advantages of intelligent sensors in such a way that the data output by the sensors can be used for further marking-based processes.

[0008] The task is solved, according to a first aspect, by a computer-implemented method for determining the marker of an object using a sensor, where the sensor is designed to scan its environment. The sensor can be an intelligent or smart sensor. All steps of the method can be performed using the sensor and / or its components.

[0009] The procedure includes:

[0010] - Generating a raw data set based on the sampled environment, wherein the raw data set characterizes a reflectivity of at least one object in the sampled environment;

[0011] - Assigning the object to a predetermined object group;

[0012] - Providing a tag database that characterizes the expected reflectivity of the object group;

[0013] - Determining a marking status of the associated object based on a comparison of the reflectivity of the associated object with the expected reflectivity of the associated object group, where the marking status characterizes the presence of a marking on the associated object;

[0014] - Output of a processed data record, wherein the processed data record at least characterizes the associated object and includes the marking status of the associated object. ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28

[0015] The proposed method allows the sensor, particularly in the form of an intelligent sensor, to be used advantageously by utilizing its computing capacity for grouping and determining whether a marker is present. By extending the data set to include the marker status, it can be used to distinguish between marked and unmarked objects. Particularly in the field of at least partially autonomous vehicles, a vehicle equipped with such a sensor can use the data set to determine which of the many vehicles in the vicinity is a marked vehicle. The method can also be advantageously applied in agriculture, construction, and production systems, where at least partially autonomous devices operate based on a scanned and, in particular, marked environment.

[0016] A particularly advantageous application of the method according to the invention is in agriculture, specifically in the leader-follower concept. The leader-follower concept describes a system in which agricultural vehicles, especially tractors or harvesters, operate in a formation. One vehicle assumes the role of the leader, while one or more vehicles act as followers. These followers are often at least partially autonomous and precisely follow the leader to efficiently and synchronously carry out work processes such as plowing, sowing, fertilizing, or harvesting. Accordingly, it is crucial that the vehicles can precisely determine which vehicle to follow.The proposed method allows for the advantageous use of intelligent sensors, as the processed data set includes the marking status, enabling vehicles to determine whether they are marked and therefore part of the formation. Consequently, agricultural vehicles can be equipped with appropriate sensors and / or reflectors. Similarly, agricultural trailers, particularly those used for seed delivery or crop transport, can also be equipped with appropriate reflectors or sensors. (See ZF Friedrichshafen AG, File 306016, Friedrichshafen, November 28, 2024.)

[0017] For the purposes of the present invention, these could also be understood as agricultural vehicles.

[0018] The sensor can be a radar sensor, a lidar sensor, a camera, and / or a stereo camera. The raw data set can include distance information, reflections, 3D points, timestamps, angle information, brightness, coordinates, and / or positions. The raw data can be grouped into objects based on various criteria and assigned to an object group. The raw data can be very large and may not contain any information about the semantic meaning or grouping of objects. Grouping or assigning an object group can refer to the classification of the objects.

[0019] With such sensors, objects in the environment that are sampled from the raw data can be identified as targets, 3D points, or 3D point clouds. These objects can then be further categorized into groups such as pedestrians, vehicles, etc.

[0020] The expected reflectivity can be based on an effective radar cross section (RCS) and / or be predetermined for a group of objects.

[0021] An object can have a natural reflectivity based on its surface properties. The reflectivity can be increased by a marker. Such a marker can include a radar reflector, a laser reflector, or similar devices. The marker can be a lidar or radar reflector.

[0022] The reflectivity of the object can be based on an interaction between the sensor's output power, for example, transmission power or optical power, and the object's reflectivity or reflectivity.

[0023] The object group can be a group of people, animals, vehicles, cars, motorcycles, buses, bicycles, trees, fences, traffic signs, or the like (ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28). The object group can be characterized by a size, in particular an area, that is characteristic of this group.

[0024] Providing the marking database can include or consist of storing it. The marking database can be stored on the sensor's memory. The stored marking database can then be accessed accordingly.

[0025] The output of the processed data set can be sent to a downstream and / or data-using unit, which determines one or more further signals based on the marking status. In the automotive sector, this unit could be the vehicle's control system, which determines whether to follow a vehicle ahead, based on whether it is a marked vehicle or not.

[0026] The processed dataset can contain further information about the object. This additional information can include: the object's position, the associated object group, the object's movement status (dynamic or static), its orientation, semantic information (contextual information such as road markings, traffic signs, lane markings), tracking information (object movement paths), and / or the object's distance. The information can vary depending on the subsequent processing of the dataset by the downstream unit, in order to minimize the processing power required by that unit. The dataset can be significantly reduced in size compared to the raw data, as it only contains the processed and interpreted information. This makes the dataset more compact and directly usable in the target application.

[0027] The comparison database can further characterize a marker identifier of the expected reflectivity of the predetermined object group. Determining the marker status can further include determining the marker identifier of the assigned object based on the comparison, as per ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28

[0028] The marker status includes the marker identifier of the assigned object. The marker identifier can be a marker ID or other identifiers. Such a marker identifier can be assigned to a predefined RCS value.

[0029] The previously and subsequently described characteristics regarding the assignment of a raw data object to an object group can be executed for a large number of raw data objects to one or more object groups. Furthermore, the described steps for a single object in the raw data can be executed for a large number of objects in the raw data.

[0030] The comparison database can characterize different expected reflectivities for a large number of object groups. Accordingly, it can be determined for each of these object groups whether an object within that group has a marker.

[0031] The comparison database can further characterize different marker identifiers for the expected reflectivities of the numerous object groups. Accordingly, different expected reflectivities, and thus different marker identifiers, can be assigned to the object groups.

[0032] The expected reflectivity of a given group of objects can be predetermined based on one or more sensor parameters and / or sensor type. Alternatively or additionally, the expected reflectivity of a given group of objects can be predetermined based on the type, size, and / or reflective surface, in particular the effective back-reflecting area, of the marker. The effective back-reflecting area can be based on the marker size and the sensor's output frequency for scanning the environment.

[0033] The expected reflectivity of an object class can correspond to the object's normal reflectivity. In other words, the expected reflectivity can be the same as that of the object class if the object had no mark and consequently no enhanced reflectivity. ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28

[0034] The presence of the assigned object's marker can be determined based on the difference between the assigned object's reflectivity and the expected reflectivity of the assigned object group, such that the marker is considered present if the difference value exceeds a predetermined threshold. The difference value can be the absolute value of the difference. The object group "human" might have an expected reflectivity or reflectivity value of RCS 1. A person in the scanned environment, for example, a pedestrian wearing a safety vest, might have a reflectivity of RCS 9. For instance, the threshold might be a reflectivity of RCS 3. In this case, the difference value would be RCS 8, which is significantly above the threshold. The threshold can be at least partially different for different object groups.The threshold value can be predefined and / or adjusted by a user. This can be done by entering the corresponding user input via a user interface of the sensor or the device mentioned below.

[0035] The marking status can further include the difference value and / or a probability based on the difference value of the marking identifier of the assigned object. Reflectors can be assigned corresponding RCS values ​​or expected reflectivities. In the previous example, with a difference value of 8, there is an 80% probability that the object is marked with an RCS-10 marking identifier. Additional information, such as whether an object is marked multiple times with reflectors, can be considered in the procedure and in calculating the probability of a marking being present. The marking status can indicate this probability.

[0036] Comparing the reflectivity of the assigned object with the expected reflectivity of the assigned object group can further include determining a sum, an average, and / or a maximum value of reflectivity values ​​of the assigned object as the reflectivity of the assigned object. ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28

[0037] According to a second aspect, the task is solved by a sensor for determining the marker of an object. The sensor can be an intelligent sensor. The sensor includes a scanning unit for scanning an environment and generating a raw data set based on the scanned environment, where the raw data set characterizes the reflectivity of at least one object in the scanned environment. Furthermore, the sensor includes a processor configured to assign the object to a predetermined object group. The sensor includes a memory configured to provide a marker database that characterizes the expected reflectivity of the object group. The processor is further configured to:

[0038] - Determining a marking status of the associated object based on a comparison of the reflectivity of the associated object with the expected reflectivity of the associated object group, where the marking status characterizes the presence of a marking on the associated object;

[0039] - Output of an edited data record, wherein the edited data record at least characterizes the associated object and includes the marking status of the associated object.

[0040] The memory can be configured to store information, in particular the raw data set and the processed data set. Furthermore, the sensor can include a communication unit for sending and / or receiving information. The communication unit can be configured to output the processed data set to a downstream unit, such as a vehicle control system.

[0041] The sensor can be configured to execute the steps of the procedure according to the first aspect. For this purpose, the procedure can be stored in memory as a computer program and be executable by the sensor.

[0042] The task is solved according to a third aspect by a device comprising a sensor according to the second aspect, wherein the device is configured to receive and evaluate the processed data set. ZF Friedrichshafen AG file 306016 Friedrichshafen 2024-11-28

[0043] The device generates one or more additional data sets and / or signals based on the evaluation of the processed data set.

[0044] The device can be a first vehicle capable of at least partial autonomous driving. The device can include a control system designed to:

[0045] - Receiving the processed data set;

[0046] - Determining the assigned object as a second vehicle marked by means of the marker based on the marker status of the assigned object;

[0047] - Issuing at least one control instruction to control the first vehicle based on the second vehicle identified by the marker.

[0048] Accordingly, the first vehicle can be controlled based on the control instruction.

[0049] The control instruction can designate the second vehicle as the lead vehicle and enable it to follow the lead vehicle at least partially autonomously. Particularly in agriculture, this allows for precise tracking of formations of multiple vehicles.

[0050] Based on the teachings described in the present invention, a person skilled in the art will recognize that the proposed method can also be used in other fields. If the sensor is an intelligent camera, it can, for example, assign object groups to containers based on a predetermined catalog or a predetermined barcode encoding and further indicate the marking status and / or the marking identifier.

[0051] The sensor's computing power is advantageously used to process the raw data.

[0052] The task is solved according to a fourth aspect by a computer program product comprising commands that cause a sensor according to the second aspect and / or a device according to the third aspect to carry out the method according to the first aspect.

[0053] Process features described with respect to the process according to the first aspect can be implemented as design features of the sensor according to the second aspect and / or of the device according to the third aspect and vice versa.

[0054] Preferred embodiments are explained by way of example with reference to the accompanying figures. These show:

[0055] Fig. 1 shows a schematic representation of a computer-implemented method for determining a marking of an object using a sensor;

[0056] Fig. 2 shows a schematic representation of a sensor for determining a marking on an object;

[0057] Fig. 3 shows a schematic representation of a device comprising such a sensor; and

[0058] Fig. 4 shows a schematic representation of a leader-follower situation based on the implementation of the procedure.

[0059] Fig. 1 shows a schematic representation of a computer-implemented method 100 for determining a marking on an object using a sensor 200. The method 100 can be stored in memory as a computer program and executed by the sensor 200 and / or a device 300 mentioned below. The sensor 200 is configured to scan the area around the sensor 200.

[0060] Method 100 comprises generating 110 a raw data set based on the sampled environment, wherein the raw data set characterizes the reflectivity of at least one object in the sampled environment. ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28

[0061] In a further step, method 100 includes assigning 120 the object to a predetermined object group. Method 100 further includes providing 130 a marker database that characterizes the expected reflectivity of the object group. It is clear to those skilled in the art that the sequence of steps shown in Fig. 1 is purely exemplary and that method 100 is not limited to this sequence. For example, step 130 can be executed first.

[0062] Procedure 100 further includes determining 140 a marking status of the assigned object based on a comparison of the reflectivity of the assigned object with the expected reflectivity of the assigned object group. The marking status characterizes the presence of a marking on the assigned object.

[0063] In addition, the procedure 100 includes outputting a processed data set, for example to a further processing unit such as a control unit of a vehicle 300, wherein the processed data set at least characterizes the associated object and includes the marking status of the associated object.

[0064] The Sensor 200 can therefore be described as an intelligent or smart sensor, since its processing power is used to assign objects to object groups and further determine whether these objects are tagged or not. The processed data set can also characterize the object's position, the distance between the sensor and the object, the object's coordinates, and its object group. Such a processed data set greatly simplifies subsequent data processing, as tagged objects can be identified quickly and easily, and their positions, for example, are known. This allows, for instance, a control instruction for the vehicle 300 to be determined more quickly and with less strain on computing resources.

[0065] Fig. 2 shows a schematic representation of a sensor 200 for performing the

[0066] Method 100. The sensor 200 includes a scanning unit for scanning the

[0067] Environment. If, for example, sensor 200 is a lidar sensor, ZF Friedrichshafen AG file 306016 Friedrichshafen 2024-11-28

[0068] The scanning unit includes a laser and a corresponding light-sensitive chip for capturing reflected light rays.

[0069] The sensor 200 further comprises a processor 220 and a memory 230. The memory 230 is designed to store information, in particular to provide a marker database that characterizes the expected reflectivity of the object group. The computer program product can be stored on the memory.

[0070] The processor 220 is configured to assign an object to a predetermined object group. Furthermore, the processor 220 is configured to determine the marking status of the assigned object based on a comparison of the reflectivity of the assigned object with the expected reflectivity of the assigned object group. The marking status indicates the presence of a marking on the assigned object. The processor 220 is also configured to output a processed data record, wherein the processed data record characterizes at least the assigned object and includes the marking status of the assigned object.

[0071] Fig. 3 shows a schematic representation of a vehicle 300, here an agricultural vehicle 300. The vehicle 300 is designed for at least partially autonomous, and in particular fully autonomous, driving. Furthermore, the vehicle 300 includes at least one such sensor 200.

[0072] Fig. 4 shows a schematic representation of an advantageous implementation of method 100 in the agricultural sector. The vehicle 300 includes the sensor 200 and is an agricultural vehicle that is at least partially autonomous.

[0073] According to Fig. 4, a road S with two lanes is shown on the left and a field F on the right. A second vehicle 400, here an agricultural vehicle with a marker, for example a reflector with RCS 10, is driving in field F. The agricultural vehicle 400 can be controlled, for example, by a user (ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28). A third vehicle, here a truck 500, is moving on road S.

[0074] The area detectable by sensor 200 is defined by the dotted line. As shown in Fig. 4, sensor 200 detects both the agricultural vehicle 400 and the truck 500 in the surrounding area. The agricultural vehicle 400 and the truck 500 may have a similarly sized reflective surface, meaning that both vehicles could, for example, be assigned to the same object group. If only this information were output, vehicle 300 would not be able to determine which vehicle to follow.

[0075] The expected reflection for this group of objects can correspond to an RCS value of 4, since trucks and agricultural vehicles are generally larger and therefore have a larger reflective surface. Furthermore, agricultural vehicle 400 has a marking, so its reflectivity is significantly increased. By comparing the expected reflection with the actual reflection of agricultural vehicle 400 and truck 500, sensor 200 can determine that truck 500 is unmarked and agricultural vehicle 400 is marked. Accordingly, based on the processed data from sensor 200 indicating the marking status, the control system for vehicle 300 can conclude that agricultural vehicle 400 is the vehicle to follow for further processing of field F. This prevents vehicle 300 from mistakenly following car 500.

[0076] By creating the raw data set and processing it accordingly using the sensor, valuable computing resources in the data processing of downstream units can be conserved and / or used for other purposes, since the processed data set characterizes the object, the group of objects, and especially the marking status of the object. ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28

[0077] Reference mark

[0078] 100 Computer-implemented method for determining a marking of an object using a sensor

[0079] 110 Generating a raw data set

[0080] 120 Assigning the object to a predetermined object group

[0081] 130 Providing a marking database

[0082] 140 Determining a marking status of the assigned object

[0083] 150 Outputting an edited data record

[0084] 200 Sensor

[0085] 210 scanning units

[0086] 220 processor

[0087] 230 storage

[0088] 300 vehicles

[0089] 400 second vehicle

[0090] 500 third vehicle

[0091] F field

[0092] S Street

Claims

ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28 Patent claims 1. Computer-implemented method (100) for determining a marking of an object by means of a sensor (200) designed to scan an environment, the method (100) comprising: Generating (110) a raw data set based on the sampled environment, wherein the raw data set characterizes a reflectivity of at least one object in the sampled environment; Assigning (120) the object to a predetermined object group; Providing (130) a tag database that characterizes an expected reflectivity of the object group; Determining (140) a marking status of the associated object based on a comparison of the reflectivity of the associated object with the expected reflectivity of the associated object group, wherein the marking status characterizes the presence of a marking on the associated object; Output (150) of a processed record, wherein the processed record at least characterizes the associated object and includes the marking status of the associated object.

2. Method (100) according to claim 1, wherein the comparison database further characterizes a marker identifier of the expected reflectivity of the predetermined group of objects, wherein determining (140) the marker status further comprises determining the marker identifier of the associated object based on the comparison, wherein the marker status comprises the marker identifier of the associated object.

3. Method (100) according to claim 1 or 2, wherein the comparison database characterizes different expected reflectivities for a plurality of object groups. ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28 4. Method (100) according to claim 3, wherein the comparison database further characterizes different marking identifiers of the expected reflectivities of the plurality of object groups.

5. Method (100) according to any of the preceding claims, wherein the presence of the marking of the associated object is determined based on a difference in the reflectivity of the associated object and the expected reflectivity of the associated group of objects, such that the marking is deemed to be present if a difference value of the difference is greater than a predetermined threshold.

6. Method (100) according to claim 5, wherein the marking status further comprises the difference value and / or a probability based on the difference value of the presence of the marking identifier of the associated object.

7. Method (100) according to any of the preceding claims, comprising comparing the reflectivity of the associated object with the expected reflectivity of the associated group of objects: Determining a sum, average and / or maximum value of reflectivity values ​​of the assigned object as the reflectivity of the assigned object.

8. Sensor (200) for determining a marker of an object, comprising: a scanning unit (210) for scanning an environment and generating a raw data set based on the scanned environment, wherein the raw data set characterizes a reflectivity of at least one object in the scanned environment; a processor (220) configured to assign the object to a predetermined object group; a memory (230) configured to provide a marker database that characterizes an expected reflectivity of the object group; wherein the processor (220) is further configured to: ZF Friedrichshafen AG File 306016 Friedrichshafen 2024-11-28 Determining a marking status of the associated object based on a comparison of the reflectivity of the associated object with the expected reflectivity of the associated object group, wherein the marking status characterizes the presence of a marking on the associated object; Output of a processed data record, wherein the processed data record at least characterizes the associated object and includes the marking status of the associated object.

9. Device (300) comprising a sensor (200) according to claim 8, wherein the device (300) is configured to receive and evaluate the processed data set.

10. Device (300) according to claim 9, wherein the device (300) is a first vehicle that is at least partially autonomous, wherein the device (300) comprises a controller configured to: receive the processed data set; Determining the assigned object as a second vehicle marked by means of the marker based on the marker status of the assigned object; Issuing at least one control instruction to control the first vehicle based on the second vehicle identified by the marker.

11. Device (300) according to claim 10, wherein the control instruction characterizes the designation of the second vehicle as the lead vehicle and at least partially autonomous following of the lead vehicle.

12. Computer program product comprising commands that cause a sensor (200) according to claim 8 and / or a device (300) according to any one of claims 9 to 11 to execute the method (100) according to any one of claims 1 to 7.

Images