Method, device, and system for automatically annotating road-related pixels in image

Abstract

Provided are a method, device, program, and storage medium for improving automatic generation of an annotated image. The method includes a step for acquiring an image, and the image includes a road. Position data of a camera that has generated the image is acquired. The coordinates of road lane boundaries are acquired. The road lane boundaries are projected onto the image on the basis of the coordinates of the road lane boundaries and the position data of the camera. The annotated image is generated by annotating, as a drivable road, each pixel of the image within the projected road lane boundaries. The annotated image is provided for additional processing.

Description

Method and apparatus for automatic annotation of road-related pixels in an image

[0001] The present disclosure relates to the annotation of road surfaces in images, and more particularly, to automated road-related pixel annotation methods and apparatuses.

[0002] Detecting drivable road surfaces in images is an important task for various applications, including autonomous vehicles, traffic management systems, and urban planning. However, developing accurate drivable road surface detection algorithms often relies on supervised machine learning models such as neural networks and requires a significant amount of annotated training data. Annotating drivable road surfaces in images poses significant challenges and can hinder the development of effective models.

[0003] One major challenge is the variability of road surfaces. Roads appear in different shapes, colors, and textures depending on factors such as materials (e.g., asphalt, concrete, gravel), weather conditions (e.g., rain, dry, snow), and environmental lighting (e.g., bright daylight, twilight, night). Capturing and correctly labeling these variations across diverse geographical locations and scenarios is a labor-intensive task.

[0004] Another problem arises from complex visual scenes. Images of road lanes often contain occlusions caused by vehicles, pedestrians, or vegetation. Road markings such as lanes and crosswalks, as well as adjacent areas such as curbs, sidewalks, or unpaved shoulders, further complicate distinguishing the road surface from other elements in the scene. Accurate annotation in such cases requires human annotators with advanced skills and consumes significant time and resources.

[0005] Furthermore, subjective interpretations and manual labeling inconsistencies can affect the quality of annotated data. Annotators may have different understandings of where the road surface begins and ends, especially in boundary cases such as partially visible roads or poorly defined roads. Such inconsistencies can reduce the reliability of the dataset and potentially result in suboptimal performance of neural networks.

[0006] Finally, the process of annotating road surfaces in images is not easily scalable. High-resolution images captured from various camera systems require meticulous attention to detail, making manual annotation both time-consuming and costly.

[0007] Furthermore, current automated annotation tools often lack the precision necessary to generate high-quality labels without significant manual intervention.

[0008] These challenges highlight the need for innovative solutions that streamline the annotation process, improve labeling accuracy, and ensure the generation of robust training datasets for neural networks. Overcoming these issues is crucial to improving the state of road surface detection and enabling more reliable deployments in real-world applications.

[0009] Therefore, an automated annotation process is needed that is efficient and improves labeling accuracy, enabling the efficient generation of training data for road detection neural networks.

[0010] In consideration of the foregoing, this disclosure provides a method, apparatus, and program for annotating drivable road surface pixels in an image, as well as a computer-readable storage medium.

[0011] According to one aspect of this disclosure, a method is provided for annotating pixels in an image corresponding to a drivable road. An image can be acquired (e.g., received). The image may include a road or at least a portion of a road. Location data of the camera that generated the image can be acquired (e.g., received). Coordinates (global coordinates) of road lane boundaries can be acquired (e.g., received or extracted). Acquisition of road lane boundary coordinates may be based on camera location data. Road lane boundary coordinates may be acquired from high-resolution (HD) map data. Road lane boundaries may be projected onto the image based on lane boundary coordinates and camera location data. An annotated image may be generated by annotating each pixel of the image within the projected road lane boundaries as a drivable road (e.g., marking or adding metadata). The annotated image may be provided for further processing.

[0012] By using precise coordinates of road lane boundaries, image data, and associated positional data, drivable road surfaces can be efficiently and reliably annotated within images.

[0013] In some embodiments, providing annotated images for further processing includes providing annotated images as training data for a drivable road detection neural network.

[0014] By using an automated process to generate training data for neural networks, neural networks can be trained on a large amount of reliable training data.

[0015] In some embodiments, the camera may be part of a road vehicle or part of a roadside entity.

[0016] In some embodiments, camera position data may include the camera's global coordinates, camera orientation, and camera height. The camera height may be indicated directly or via its relative height to a corresponding vehicle or roadside entity.

[0017] In some embodiments, projecting road lane boundaries onto an image based on the coordinates of the road lane boundaries and camera position data may include determining a (virtual) image plane based on the camera position data. The virtual image plane may correspond to the viewpoint of the image. Furthermore, road lane boundaries can be projected onto the (virtual) image plane based on the camera position data and the coordinates of the road lane boundaries.

[0018] In some embodiments, projecting road lane boundaries onto the image plane is further based on the radial distortion of the camera. Furthermore, other distortions that affect the projection of road lane boundaries onto the image plane may also be considered.

[0019] In some embodiments, determining the (virtual) image plane based on camera position data may be further based on the camera's focal length (i.e., the focal length of the lens system included in the camera).

[0020] In some embodiments, annotating each pixel of an image within a projected road lane boundary as a drivable road includes annotating a pixel of an image as a drivable road if the pixel lies between two projected road lane boundaries of the same road lane.

[0021] In some embodiments, annotating each pixel of the projected road lane boundary image as a drivable road may further include determining obstacles on the drivable road. Furthermore, each pixel corresponding to an obstacle may be excluded from the annotated pixels (e.g., before or after the first annotation).

[0022] In some embodiments, determining obstacles on a drivable road may include a depth estimation algorithm. The depth estimation algorithm may include determining the estimated depth of an annotated pixel (an estimate of the distance of the object to the camera) based on the image. Furthermore, the predicted depth of an annotated pixel may be determined based on projected road lane boundaries (an estimate of the distance from the camera to the road surface without objects / obstacles). The estimated depth may be compared to the predicted depth to determine whether or not obstacles are present on the drivable road. The determination of whether or not obstacles are present on the drivable road may be based on a threshold, for example, an adjustable threshold. In other words, if the difference between the estimated distance to the road surface (without obstacles) and the estimated distance to the object exceeds a threshold, it may be determined that obstacles are present on the drivable road surface.

[0023] By including obstacle detection based on depth estimation, obstacles can be automatically annotated within the image of the drivable road surface, even if they are located on the drivable road surface.

[0024] In some embodiments, excluding each pixel corresponding to an obstacle from annotated pixels may include designating a safety zone around the object and excluding the corresponding pixels so as not to be annotated as a drivable road surface. Furthermore, excluding each pixel corresponding to an obstacle from annotated pixels may be based further on the estimated height and depth of the obstacle.

[0025] In some embodiments, determining the estimated depth may further depend on a second image from a second camera. The second image and the image can form a stereo image. Alternatively, determining the estimated depth may further depend on LIDAR data of the environment captured in the image.

[0026] Aspects of this disclosure may be implemented via a device. The device may include a processor and memory coupled to the processor. The processor may be adapted to perform the methods according to the aspects and embodiments of this disclosure.

[0027] In some embodiments, the device may be part of a server, a road vehicle, or a roadside entity.

[0028] Aspects of this disclosure may be implemented via a system. The system may include a cloud infrastructure, including apparatus for performing the methods according to the aspects and embodiments of this disclosure. The system may further include a road vehicle, including a camera for generating images and corresponding location data. The road vehicle may transmit the images and the location data from the camera that generated the images to the cloud infrastructure.

[0029] Aspects of the present disclosure may be implemented via a program. Once the instructions of the program are executed by a processor, the processor can perform the aspects and embodiments of the present disclosure. Computer-readable storage media can store programs. Such computer-readable storage media may include, but are not limited to, memory devices such as random access memory (RAM) devices and read-only memory (ROM) devices described herein. Accordingly, some innovative aspects of the subject matter described herein may be implemented via one or more computer-readable storage media on which software is stored.

[0030] It will be understood that the features of the apparatus and the steps of the method may be interchangeable in various ways. In particular, as those skilled in the art will understand, the details of the disclosed method(s) may be implemented by the corresponding apparatus(s) (or systems), and vice versa. Furthermore, it will be understood that any of the above descriptions relating to the method(s) apply equally to the corresponding apparatus(s) (or systems), and vice versa.

[0031] Exemplary embodiments of this disclosure are described below with reference to the accompanying drawings.

[0032] This flowchart shows an example of a method for annotating road-related pixels in an image according to embodiments of the present disclosure. (a) to (c) are schematic diagrams illustrating a method for determining drivable road surfaces in an image using an image and HD-Map information according to embodiments of the present disclosure. This flowchart shows an example of a method for determining obstacles on a drivable road surface according to embodiments of the present disclosure. This diagram schematicly illustrates the determination of the expected depth of pixels for determining obstacles on a drivable surface according to embodiments of the present disclosure. This diagram schematicly illustrates an example of an apparatus for annotating road-related pixels in an image according to embodiments of the present disclosure. This diagram schematicly illustrates an example of a system for annotating road-related pixels in an image according to embodiments of the present disclosure.

[0033] The figures (multiple figures) and the following description relate to preferred embodiments for illustrative purposes only. It should be noted that from the following description, alternative embodiments of the structures and methods disclosed herein will be readily recognizable as viable alternatives that can be used without departing from the principles of the claims.

[0034] Herein, several embodiments are given in detail, examples of which are shown in the accompanying drawings. Note that, to the extent possible, similar or similar reference numerals may be used in the drawings to indicate similar or similar functions. The drawings depict embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein can be used without departing from the principles described herein.

[0035] Training a neural network to detect drivable road surfaces requires a large amount of various training data, namely, images containing roads and corresponding images annotated with drivable road surfaces, for example, by adding metadata that can identify each pixel corresponding to a drivable road surface.

[0036] Regarding the automatic generation of the aforementioned annotated images, the map data was far too inaccurate to be used to determine road markings such as road lane boundaries within images showing roads.

[0037] However, with the advent of HD maps, it may be possible to achieve sufficient accuracy to use the coordinates of road markings for automatic annotation of drivable road surfaces within images.

[0038] HD maps are designed to meet the precision requirements of advanced navigation and autonomous driving systems. Unlike conventional maps used for human navigation, HD maps provide the high level of detail, precision, and contextual information essential for machine-based interpretation.

[0039] These maps go beyond basic road layouts and landmarks, incorporating accurate 3D representations of dynamic elements such as roads, lanes, curbs, road markings, traffic signs, and even signal phases. With centimeter-level accuracy, HD maps are a crucial component in many applications.

[0040] By leveraging data from LiDAR, radar, cameras, and other sensor technologies, HD maps are constructed as multi-layered representations. Specifically, HD maps include a geometry layer for accurate spatial measurement of roads and their features. These road features, such as road lane boundaries, are assigned global coordinates. Below, these precise coordinates of road-level boundaries are used to automatically annotate pixels in the image that relate to drivable road surfaces.

[0041] Figure 1 illustrates a flowchart of an exemplary method 100 for annotating pixels related to drivable road surfaces in an image.

[0042] In step S102, an image is acquired. The image shows at least a portion of the road. The image may be received from a remote entity that includes a camera for capturing the image. The remote entity may be a road vehicle or a roadside entity. Alternatively, the image may be received from a server that stores road-related images.

[0043] In step S104, the position data of the camera that captured the image is acquired. The position data of the camera may correspond to the position data of the camera at the moment the image was captured. When the image is received from a remote entity or server, the position data may be attached to the image as metadata. The position data can include the coordinates of the camera. The coordinates may be global coordinates, for example, GPS coordinates.

[0044] The position data can further include the height of the camera, that is, the distance to the ground. Alternatively, the height of the camera may be fixed or known. For example, the camera may be installed at a certain height within a road vehicle or roadside infrastructure. In another example, the height of the road vehicle and the relative position of the camera within the road vehicle may be known. In these cases, it is not necessary to transmit the camera height information.

[0045] The position data can further include the orientation (direction) of the camera. The orientation of the camera may be provided, for example, in the form of three rotation angles (pitch, yaw, roll). Alternatively, other formats may be used to represent the orientation of the camera. The orientation of the camera may be generated by the camera's sensor or by the sensors of the road vehicle / roadside entity equipped with the camera.

[0046] In addition to the position data of the camera, additional camera data may be received. Such data can include, for example, the radial lens distortion of the camera lens.

[0047] In step S106, the coordinates of the road lane boundaries are acquired. The coordinates may be global coordinates. The accuracy of the coordinates may be within the centimeter range. In other words, the coordinates accurately describe the actual environment, such as the road lane boundaries.

[0048] The coordinates of the road lane boundaries may be received from an HD map, for example, from a server that stores HD map data. Further, additional data related to the road, for example, any data that can be used to determine the drivable road surface, may be acquired from the HD map.

[0049] The coordinates of road lane boundaries may be obtained based on camera position data. For example, the coordinates of all road lane boundaries within a specific radius around the camera's global coordinates can be obtained. Alternatively, the coordinates of all road lane boundaries within a triangle on the HD map can be obtained. The triangle can be determined based on the camera's coordinates and orientation.

[0050] In step S108, the road lane boundary is projected onto the image based on the coordinates of the road lane boundary and the camera's position data. In particular, when the road lane boundary is represented as a number of points, each with its own coordinates, each point is projected onto the image (if the projection of a point is within the image).

[0051] Projecting road lane boundaries onto an image based on the coordinates of the road lane boundaries and the camera's position data may involve determining the camera plane based on the camera's position data. The camera plane (virtual image plane) may be a virtual plane in 3D space (e.g., absolute coordinates) and can describe how the camera "sees" the scene, i.e., the actual live scene captured on the image. Depending on the camera's orientation, the camera plane may be positioned in front of the camera at a distance corresponding to the camera's focal length.

[0052] Furthermore, radial distortion of the camera (i.e., the camera's lens system) can be taken into account, for example, when determining the virtual image plane by applying the corresponding distortion function to the coordinates of the virtual image plane or road lane boundary. This step may be particularly important when the camera uses a so-called fisheye lens.

[0053] After determining the camera plane, the road lane boundaries can be projected onto the camera plane based on the camera's position data and the lane boundary coordinates. In particular, a ray may be projected from the camera's coordinates to the coordinates of a point on the road lane boundary. The intersection of this ray and the camera plane then corresponds to the projection of the road lane boundary point onto the camera plane. Similarly, pixels corresponding to the intersection on the image can represent the projection of the road lane boundary point onto the image.

[0054] The process of projecting road lane boundaries onto an image can also be illustrated in Figures 2(a)–(c). These figures show the scene in 2D for illustrative purposes, but the projection is performed in 3D. Figure 2(a) depicts the road lane boundary coordinates obtained from an HD map. In Figure 2(b), camera coordinates are added to the 2D / 3D space along with the camera orientation. The next step is to add a camera plane (virtual image plane) (illustrated by dashed triangles). Finally, in Figure 2(c), for each point of the road lane boundary, a ray is projected from the camera coordinates to the road lane boundary coordinates (illustrated by two exemplary gray rays). The intersections of the gray rays and the dashed triangles can represent the projection of the road lane boundary onto the camera plane / image. By projecting all points of the road lane boundary onto the camera plane, the road lane boundary can be precisely determined within the image.

[0055] Next, in step S110, an annotated image is generated by annotating each pixel of the projected image within the road lane boundary as a drivable road. In particular, all pixels may be annotated if they are between two road lane boundaries of the same road lane.

[0056] Pixel annotations can be understood as adding metadata to an image that describes whether each pixel represents a drivable surface. Alternatively, it is possible to generate a binary image where each pixel representing a drivable surface is represented by 0 and the remaining pixels by 1, and vice versa. Or, pixels corresponding to drivable surfaces can be marked with a specific color. The color may be transparent, meaning the original image remains visible. The form of the annotation may depend on the application using the annotated image.

[0057] This process allows for the automatic annotation of drivable road surfaces within images without the need for human supervision. Furthermore, the annotations are potentially highly reliable because they are independent of image quality, road quality, and weather conditions.

[0058] In the final step S112, the annotated image is provided for further processing. A specific relevant field for using the annotated image may be training a neural network for detecting drivable surfaces. In particular, the unannotated image may be the input to the neural network, and the annotated image may resemble ground truth. This allows for efficient training of the neural network for detecting drivable surfaces.

[0059] However, the use of annotated images is not limited to training neural networks; it can also be used for other automated tasks. For example, roadside entities could detect when a vehicle has left the drivable road surface based on annotated images and act automatically, i.e., by sending a warning to each vehicle.

[0060] While the process of automatically annotating pixels can work well for any road structure and any weather conditions, images often also include obstacles on the road, such as people. Since obstacles on the road should be avoided by autonomous vehicles, the training data for the drivable road detection neural network also needs to exclude obstacles from the drivable road surface. To achieve this, an obstacle detection method can be added to the annotation method 100.

[0061] Figure 3 illustrates a flowchart of an exemplary method 200 for detecting obstacles on a drivable road surface in an image. Method 200 may be performed after pixels have already been annotated, and the annotations may be removed depending on the result of method 200, or method 200 may be performed as part of step S110, i.e., pixels are excluded from annotation based on the result of method 200. The first case is described below.

[0062] In step S202, the estimated depth of the annotated pixels is determined based on the image. In particular, the depth can be estimated for each annotated pixel, that is, for each pixel determined to correspond to a drivable road surface. Alternatively, a method for detecting objects in the image, such as a neural network, can be used. Then, the depth is estimated only for pixels that correspond to a drivable road surface and also correspond to an object. In other words, depth estimation is performed only for pixels that correspond to a drivable road surface that is obscured by an obstacle.

[0063] The depth estimation of each pixel can be performed by one of the following three methods:

[0064] Image-only depth estimation relies on a single 2D image as input and leverages machine learning, particularly deep neural networks, to estimate depth. Techniques such as convolutional neural networks (CNNs) are trained on large datasets to recognize depth cues such as object size, perspective, and shading. Transformer architectures or self-supervised learning can also be used to improve accuracy without requiring depth-annotated data.

[0065] Therefore, the depth of pixels can be estimated using only existing images.

[0066] Stereo Image Depth Estimation Stereo image depth estimation mimics human binocular vision by using two images captured simultaneously from slightly different viewpoints. By analyzing the disparity between corresponding points in the images, the algorithm can directly calculate depth. Furthermore, a trained neural network can be used to improve the results.

[0067] In this case, two images may be required from two cameras positioned on a road vehicle or roadside entity to mimic human binocular vision.

[0068] Lidar-based depth estimation: Lidar (Light Detection and Ranging) systems directly measure distance by using laser pulses and calculating the time of flight of the emitted light. Lidar sensors emit laser pulses, measure the time it takes for them to return, and generate a 3D point cloud. Each point in the point cloud has (x, y, z) coordinates in the sensor's coordinate system. To map the point cloud information onto an image, the 3D points in the Lidar point cloud must be projected onto the image plane, i.e., a process similar to step S108 must be performed. In particular, the process is limited to pixels annotated as drivable road surfaces.

[0069] As a result, an image is obtained with high-precision depth data for every annotated pixel.

[0070] In step S204, the expected depth of the annotated pixels is determined based on the projected road lane boundaries. Similar to step S202, the expected depth may be calculated for all annotated pixels, or it may be calculated only for pixels corresponding to obstacles detected by the object detection algorithm.

[0071] An example of this process is illustrated in Figure 4. The distance (2D) between the camera and the lane boundary is known through the global coordinates of the road lane boundary and the camera. Furthermore, the camera height may be known, or information regarding the camera height is received, so the distance from the camera to a point on the road lane boundary can be calculated more accurately using simple geometric methods. Furthermore, since points are projected onto the image, the determined distance may be used as the expected depth of the corresponding pixel. For the remaining pixels, i.e., pixels representing the drivable road surface between the road lane boundaries, the expected depth can be determined based on the distance of the point corresponding to the pixel from the road lane boundary. Alternatively, the pixel depth of a point between road lane boundaries can be determined based on interpolation of the depth value of the pixel corresponding to the road lane boundary. In other words, it is assumed that the road is a 2D plane.

[0072] In step S206, the estimated depth and the predicted depth are compared to determine whether an obstacle exists on the drivable road. The determination of whether an obstacle exists on the drivable road can be based on a predetermined threshold; that is, if the predicted depth and estimated depth of a particular pixel are greater than the predetermined threshold, it is determined that an obstacle exists on the drivable road surface. The determination of whether an obstacle exists on the drivable road can also be based on the z-buffer.

[0073] Next, the identified obstacles must be excluded from the drivable surface. In a simple implementation, each pixel exceeding a predetermined threshold is excluded from being annotated as part of the drivable surface. Such a method may be feasible for obstacles that have a relatively low height and a relatively large depth (in the camera's field of view). Furthermore, pixels that are a predetermined distance from the excluded pixels may also be excluded from being annotated as part of the drivable surface in order to implement a safety zone around the obstacle.

[0074] For objects with relatively large heights and relatively small depths (e.g., humans), a second threshold can be defined. If the difference between the expected threshold and the estimated threshold exceeds this threshold, the pixel is not excluded from the drivable road surface. This means that tall obstacles may not affect the drivable road surface annotation far behind the obstacle.

[0075] By implementing the aforementioned obstacle detection and corresponding annotation modification, efficient automatic annotation of drivable road surfaces can be achieved even when obstacles are located on the road.

[0076] The above describes a method for annotating road surface-related pixels in an image, but this disclosure relates to a similar device. Next, an embodiment of such a device will be described with reference to Figure 5.

[0077] As shown in Figure 5, the device 300 includes a processor 301 and a memory 302. The memory 302 is configured to store program code. The processor 301 is configured to execute instructions in the program code, and as a result, the device 300 performs the pixel annotation method in any of the embodiments and implementations described above. The processor 301 can also receive appropriate input data (e.g., images, camera position data, lane boundary coordinates, etc.), among other things, depending on the use case and / or implementation. Depending on the use case and / or implementation, the processor 301 may be adapted to perform the methods / technologies described throughout this disclosure (e.g., methods 100 and 200 described above with reference to Figures 1 and 3, respectively) and generate corresponding output data (e.g., annotated images). The device may be part of a server, e.g., part of a cloud infrastructure. Alternatively, the device may be part of a road vehicle or roadside entity.

[0078] Finally, Figure 6 shows an exemplary system for annotating road-related pixels in an image. The system comprises two main components: a vehicle V (or alternatively, a roadside entity) and a cloud C (i.e., a server structure that can spread around multiple locations).

[0079] Vehicle V may be equipped with a sensor unit 10 for generating images and corresponding location data. Thus, the sensor unit 10 may include one or more cameras 11 and additional sensors that can be used for location determination 15, such as an odometer 12, an IMU 13, and / or a GNSS unit 14. Vehicle V may communicate images along with corresponding location data to cloud C via a communication interface (I / F).

[0080] Cloud C may include an HD map 20, i.e., HD map data may be available within Cloud C. From the HD map data, relevant portions (i.e., road lane boundaries) may be obtained based on location data received along with the images. A device in the cloud, e.g., device 300, can then use the available information, i.e., the HD map data of road lane boundaries, camera images, and camera location data, to perform method 100 and optionally method 200. The annotated images (labeled data) may then be stored in a storage unit 21 in Cloud C along with the original images, i.e., unannotated images, and a depth estimation unit 22 and, for example, a projection unit 21 that generates the labeled data may also be part of Cloud C. Both images can then be used to train a neural network N (which may include one or more machine learning models N1 and evaluation units N2) for detecting drivable road surfaces on the images (within the cloud). In particular, unannotated images may be used as input to a neural network N, while annotated images are used to evaluate the results of the neural network N and to fit the weights of the neural network N based on the evaluation.

[0081] This makes it possible to generate efficient training data for the drivable road surface detection neural network N.

[0082] Interpretation Unless otherwise specified, as will be apparent from the following description, any description using terms such as “process,” “calculate,” “calculate,” “determine,” and “analyze” throughout this invention is understood to refer to the operation and / or process of a computer or computing system or similar electronic computing device that manipulates and / or converts data expressed as physical quantities, such as electron quantities, into other data expressed as similar physical quantities.

[0083] Throughout this invention, any reference to “one exemplary embodiment,” “several exemplary embodiments,” or “exemplary embodiments” means that certain features, structures, or characteristics described in relation to the exemplary embodiments are included in at least one exemplary embodiment of this invention. Therefore, the occurrence of the phrases “in one exemplary embodiment,” “several exemplary embodiments,” or “exemplary embodiments” in various parts of this invention does not necessarily all refer to the same exemplary embodiment. Furthermore, certain features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments, as will be apparent to those skilled in the art.

[0084] Where used herein, unless otherwise specified, the use of ordinal adjectives such as “first,” “second,” “third,” etc., to describe a common object is merely to indicate that different instances of a similar object are being referred to, and is not intended to imply that the objects described in this manner must be in a given order in time, space, order, or any other way.

[0085] Furthermore, it should be understood that the expressions and terms used herein are for illustrative purposes only and are not intended to limit them. The use of “including,” “comprising,” or “having,” and their variations, means to include the items listed thereafter and their equivalents, as well as any additional items. Unless otherwise specified or limited, the terms “attached,” “connected,” “supported,” and “combined,” and their variations, are to be interpreted broadly and include both direct and indirect attachment, connection, support, and combination.

[0086] In the following claims and description herein, any one of the terms comprising, composed of, or which comprises is an open term meaning that it includes, but does not exclude, at least the following elements / features. Therefore, when used in the claims, the term comprising should not be interpreted as limiting to the means, elements, or steps enumerated thereafter. For example, the expression "device including A and B" should not be limited to a device consisting only of elements A and B. As used herein, any one of the terms including, which include, or that includes is also an open term meaning that it includes, but does not exclude, at least the elements / features that follow that term. Therefore, including is synonymous with comprising and means the same.

[0087] Furthermore, it should be noted that the examples in this disclosure may take the form of entirely hardware examples, entirely software examples (including firmware, resident software, microcode, etc.), or examples combining software and hardware aspects. In addition, the examples in this disclosure may take the form of computer program products on computer-readable media having computer executable program code embedded in the medium.

[0088] Note that arrows may be used in drawings to represent communication, transfer, or other activities involving two or more entities. A bidirectional arrow generally indicates that an activity can occur in both directions (e.g., a command / request in one direction and a corresponding response in the other, or peer-to-peer communication initiated by either entity), although in some situations, the activity may not necessarily occur in both directions.

[0089] While one-way arrows can generally indicate one-way activity, it should be noted that in certain situations, such one-way activity may actually include two-way activity (e.g., a message from sender to receiver and an acknowledgment from receiver to sender, or the establishment of a connection before forwarding and the termination of a connection after forwarding). Therefore, the type of arrow used in a particular drawing to represent a particular activity is illustrative and should not be considered limiting.

[0090] The embodiments / examples are described above with reference to flowcharts and / or block diagrams of methods and apparatus, etc. It will be understood that each block in the flowcharts and / or block diagrams, and / or combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer executable program code.

[0091] Computer executable program code may be provided to a processor of a general-purpose computer, a dedicated computer, or other programmable data processing device for manufacturing a particular machine, and as a result, program code executed via the processor of the computer or other programmable data processing device creates means for implementing functions / operations / outputs specified in a flowchart, block diagram block or multiple blocks, diagrams, and / or a written description.

[0092] These computer-executable program codes may also be stored in computer-readable memory that can instruct a computer or other programmable data processing device to function in a particular way, and as a result, the program code stored in computer-readable memory may produce a product that includes instruction means for implementing functions / operations / outputs specified in flowcharts, block diagram blocks, figures, and / or descriptions.

[0093] Computer-executable program code may also be loaded into a computer or other programmable data processing device such that a set of operational steps are executed on the computer or other programmable device to generate a computer implementation process, and as a result, the program code executed on the computer or other programmable device may provide steps for implementing the functions / operations / outputs specified in the flowchart, block diagram blocks, figures, and / or the description provided. Alternatively, the computer program implementation steps or operations may be combined with steps or operations implemented by an operator or human to perform the embodiment.

[0094] Communication networks can generally include public and / or private networks, and can include local area, wide area, metropolitan area, storage, and / or other types of networks, and can use communication technologies, including, but not limited to, analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth), networking technology, and internetworking technology.

[0095] It should also be noted that devices may use communication protocols and messages (e.g., messages created, sent, received, stored, and / or processed by the device), and such messages may be transmitted through communication networks or media.

[0096] Unless the context should indicate otherwise, this disclosure should not be construed as being limited to any particular communication message type, communication message format, or communication protocol. Therefore, communication messages may, in general, include, but are not limited to, frames, packets, data crams, user datagrams, cells, or other types of communication messages.

[0097] Unless the context indicates otherwise, references to specific communication protocols are illustrative, and alternative embodiments may, as needed, be variations of such communication protocols (e.g., modifications or extensions to protocols that may be made from time to time), or other known or future protocols.

[0098] Logical flows may be described herein to demonstrate various aspects, and it should be noted that this disclosure should not be construed as limiting it to any particular logical flow or logical implementation. The described logic may be divided into different logical blocks (e.g., programs, modules, functions, or subroutines) without altering the overall result.

[0099] In many cases, logical elements can be added, modified, omitted, executed in a different order, or implemented using different logical structures (e.g., logic gates, loop primitives, conditional logic, and other logical structures) without changing the overall result.

[0100] This disclosure may be implemented in many different forms, without limit, including computer program logic for use with a processor (e.g., a microprocessor, microcontroller, digital signal processor, or general-purpose computer), programmable logic for use with a programmable logic device (e.g., a field-programmable gate array (FPGA) or other PLD), individual components, integrated circuits (e.g., application-specific integrated circuits (ASICs)), or any other means including any combination thereof. Computer program logic implementing some or all of the described functions is typically translated into a computer executable format, stored as such on a computer-readable medium, and implemented as a set of computer program instructions executed by a microprocessor under the control of an operating system. Hardware-based logic implementing some or all of the described functions may be implemented using one or more appropriately configured FPGAs.

[0101] Computer program logic that implements all or part of the functions described herein may be embodied in a variety of forms, including, but not limited to, source code form, computer executable form, and various intermediate forms (e.g., forms generated by an assembler, compiler, linker, or locator).

[0102] Source code may include a set of computer program instructions implemented in one of several programming languages ​​(e.g., object code, assembly language, or high-level languages ​​such as Fortran, C, C++, Java, or HTML) for use in various operating systems or environments. Source code may define and use various data structures and communication messages. Source code may be in computer executable format (e.g., via an interpreter), or source code may be converted into computer executable format (e.g., via a translator, assembler, or compiler).

[0103] Computer executable program code for performing the operations of the embodiments of this disclosure may be written in an object-oriented, scripting, or non-scripting programming language such as Java, Perl, Smalltalk, or C++. However, computer executable program code for performing the operations of the embodiments may also be written in a conventional procedural programming language such as the C programming language or a similar programming language.

[0104] Computer program logic that implements all or part of the functions described herein may run on a single processor at different times (e.g., in parallel), or on multiple processors at the same or different times, or under a single operating system process / thread, or under different operating system processes / threads.

[0105] Therefore, the term "computer process" can generally refer to the execution of a set of computer program instructions, regardless of whether different computer processes run on the same or different processors, or whether different computer processes run under the same operating system process / thread or different operating system processes / threads.

[0106] Computer programs may be permanently or temporarily fixed to tangible storage media in any form (e.g., source code, computer executable, or intermediate), such tangible storage media include semiconductor memory devices (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), magnetic memory devices (e.g., diskettes or fixed disks), optical memory devices (e.g., CD-ROMs), PC cards (e.g., PCMCIA cards), or other memory devices.

[0107] A computer program may contain any form of a signal that can be transmitted to a computer using any of the various communication technologies, including, but not limited to, analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth), networking technology, and internetworking technology.

[0108] Computer programs may be distributed in any form as removable storage media with accompanying printed or electronic documents (e.g., shrink-wrapped software), pre-loaded onto a computer system (e.g., on a system ROM or fixed disk), or distributed from a server or electronic bulletin board via a communication system (e.g., the Internet or the World Wide Web).

[0109] Hardware logic (including programmable logic for use in programmable logic devices) that implements all or part of the functions described herein may be designed using conventional manual methods, or may be designed, captured, simulated, or documented electronically using various tools such as computer-aided design (CAD), hardware description languages ​​(e.g., VHDL or AHDL), or PLD programming languages ​​(e.g., PALASM, ABEL, or CUPL).

[0110] Any suitable computer-readable medium can be used. This computer-readable medium may, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, devices, or media.

[0111] More specific examples of computer-readable media include, but are not limited to, electrical connections having one or more wires, or other tangible storage media such as portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), compact disc read-only memory (CD-ROM), or other optical or magnetic storage devices.

[0112] The programmable logic may be permanently or temporarily fixed to a tangible storage medium such as a semiconductor memory device (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), a magnetic memory device (e.g., a diskette or fixed disk), an optical memory device (e.g., a CD-ROM), or other memory device.

[0113] The programmable logic may be fixed to a signal that can be transmitted to a computer using any of the various communication technologies, including, but not limited to, analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth), networking technology, and internetworking technology.

[0114] The programmable logic may be distributed as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink-wrapped software), pre-loaded onto a computer system (e.g., on a system ROM or fixed disk), or distributed from a server or electronic bulletin board via a communication system (e.g., the Internet or the World Wide Web). Of course, some embodiments may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments may be implemented entirely as hardware or entirely as software.

[0115] While certain exemplary embodiments are described and illustrated in the accompanying drawings, it should be understood that such embodiments are illustrative and not limited to the specific structures and arrangements illustrated and described, as various other changes, combinations, omissions, modifications, and substitutions are possible in addition to those described in the paragraphs above.

[0116] Those skilled in the art will understand that various adaptations, modifications, and / or combinations of the embodiments described above can be formed. Therefore, it should be understood that, within the scope of the appended claims, this disclosure may be implemented in ways other than those specifically described herein. For example, unless otherwise specified, the steps of the processes described herein may be performed in an order different from that described herein, and one or more steps may be combined, divided, or performed simultaneously. Those skilled in the art will also understand that, considering this disclosure, different examples or embodiments described herein may be combined to form other examples.

[0117] In the above description of exemplary embodiments of the present invention, it should be understood that, for the purpose of simplifying the invention and aiding in the understanding of one or more of the various aspects of the invention, various features of the invention may be summarized in a single exemplary embodiment, figure, or description thereof. However, this method of the present invention should not be interpreted as reflecting an intention that the claims require more features than those expressly described in each claim. Rather, as reflected in the claims below, the aspects of the invention lie in fewer features than all of the aforementioned single disclosed exemplary embodiment. Accordingly, the claims following this specification are expressly incorporated herein, and each claim stands independently as a separate exemplary embodiment of the invention.

[0118] Furthermore, while some exemplary embodiments described herein include some, but not all, of the features contained in other exemplary embodiments, it will be understood by those skilled in the art that combinations of features from different exemplary embodiments fall within the scope of the invention and form different exemplary embodiments. For example, any of the claimed exemplary embodiments can be used in any combination within the claims below.

[0119] Numerous specific details are provided in the description herein. However, it is understood that exemplary embodiments of the invention may be carried out without these specific details. In other examples, well-known methods, structures, and techniques are not described in detail so as not to obscure the understanding of this description.

[0120] Therefore, while we have described what is considered to be the best mode of the present invention, those skilled in the art will recognize that other modifications and further modifications can be made without departing from the spirit of the invention, and that all such changes and modifications are intended to be claimed to be within the scope of the invention. For example, any of the above-described methods are merely representative examples of procedures that may be used. Functions may be added or removed from the block diagram, and operations may be interchanged between function blocks. Steps may be added or removed in the methods described within the scope of this disclosure.

Claims

1. A method for annotating pixels in an image corresponding to a drivable road, comprising: acquiring an image, wherein the image includes a road; acquiring position data of a camera that generated the image; acquiring coordinates of a road lane boundary; projecting the road lane boundary onto the image based on the coordinates of the road lane boundary and the position data of the camera; generating an annotated image by annotating each pixel of the image within the projected road lane boundary as the drivable road; and providing the annotated image for further processing.

2. The method according to claim 1, wherein the step of providing the annotated image for further processing includes the step of providing the annotated image as training data for a drivable road detection neural network.

3. The method according to claim 1 or 2, wherein the coordinates of the road lane boundary are global coordinates obtained from a high-resolution HD map.

4. The method according to any one of claims 1 to 3, wherein the step of obtaining the coordinates of the road lane boundary is based on the position data of the camera.

5. The method according to any one of claims 1 to 4, wherein the position data of the camera includes the global coordinates of the camera, the orientation of the camera, and the height of the camera.

6. The method according to any one of claims 1 to 5, wherein the camera is part of a road vehicle or part of a roadside entity.

7. The method according to any one of claims 1 to 6, wherein the step of projecting the road lane boundary onto the image based on the coordinates of the lane boundary and the position data of the camera includes the steps of determining an image plane based on the position data of the camera and projecting the road lane boundary onto the image plane based on the position data of the camera and the coordinates of the road lane boundary.

8. The method according to claim 7, wherein the step of determining the image plane is further based on the radial distortion of the camera.

9. The method according to any one of claims 1 to 8, wherein the step of annotating each pixel of the image within the projected road lane boundary as the drivable road includes the step of annotating the pixel of the image as the drivable road if the pixel is between two projected road lane boundaries of the same road lane.

10. The method according to any one of claims 1 to 9, wherein the step of annotating each pixel of the image within the projected road lane boundary as the drivable road further comprises the steps of determining an obstacle on the drivable road and removing each pixel corresponding to the obstacle from the annotated pixels.

11. The method according to claim 10, wherein the step of determining an obstacle on the drivable road includes: determining an estimated depth of an annotated pixel based on the image; determining an expected depth of the annotated pixel based on the projected road lane boundary; and comparing the estimated depth with the expected depth to determine whether an obstacle exists on the drivable road.

12. The method according to claim 11, wherein the step of determining the estimated depth is further based on a second image from a second camera, wherein the second image and the images form a stereo image, or the step of determining the estimated depth is further based on LIDAR data of the environment captured in the image.

13. The method according to claim 11 or 12, wherein the step of comparing the estimated depth with the predicted depth to determine whether or not an obstacle exists on the drivable road is based on a threshold.

14. An apparatus comprising a processor and a memory coupled to the processor, wherein the processor is adapted to perform the method described in any one of claims 1 to 13.

15. The apparatus according to claim 14, comprising a server, a road vehicle, and / or a roadside entity.

Images