Sidelobe removal and clutter retention in a radar imaging pipeline

Abstract

Enhanced radar systems that can provide high-resolution radar images including clutter are disclosed. A enhanced radar system may include radar units that generate and transmit low-level radar data and a high-performance computing system that receives and processes the low-level radar to generate the high-resolution image. Performing a portion or the entire radar data processing using the high-performance computing system enables usage of advanced signal processing and machine learning techniques that allow retaining clutter information and generate high resolution radar images that include semantic target information.

Description

MOTN.153WO PATENTSIDELOBE REMOVAL AND CLUTTER RETENTION IN A RADAR IMAGING PIPELINECROSS-REFERENCE TO RELATED APPLICATIONS[OOOIJThis application claims the priority benefit of U.S. Patent Prov. App. 63 / 655441, entitled SIDELOBE REMOVAL AND CLUTTER RETENTION IN A RADAR IMAGING PIPELINE, June 3, 2024, which is incorporated herein by reference in its entirety.BRIEF DESCRIPTION OF THE FIGURES

[0002] FIG. 1 is an example environment in which a vehicle including one or more components of an autonomous system can be implemented.

[0003] FIG. 2 is a diagram of one or more example systems of a vehicle including an autonomous system.

[0004] FIG. 3 is a diagram of components of one or more example devices and / or one or more example systems of FIGS. 1 and 2.

[0005] FIG. 4A is a diagram of certain components of an example autonomous system.

[0006] FIG. 4B is a diagram of certain components of an example autonomous system.

[0007] FIG. 4C is a diagram of certain components of an example autonomous system.

[0008] FIG. 4D is a diagram of certain components of an example autonomous system.

[0009] FIG. 5A schematically illustrates an existing imaging radar system comprising digital signal processing (DSP) modules between individual radar radio frontend modules and a high-performance computing (HPC) system.

[0010] FIG. 5B schematically illustrates an imaging radar system comprising direct connections between individual radar radio frontend modules and a high-performance computing (HPC) system, according to certain embodiments disclosed herein.

[0011] FIGS. 6 schematically illustrates an example implementation of the imaging radar system shown in FIG. 5B, according to certain embodiments disclosed herein.

[0012] FIG. 7A schematically illustrates an example radar imaging pipeline that may be used in the imaging radar systems shown in FIGS. 5A-5B to generate a radar image from low-level data streams received from an individual radar unit, according to certain embodiments disclosed herein.MOTN.153WO PATENT

[0013] FIG. 7B schematically illustrates an example semantic preserving filter (SPF) arrangement that can be used to process the range Doppler cube (RDC) generated in the radar imaging pipeline shown in FIG. 7A.

[0014] FIG. 7C schematically illustrates an example semantic preserving filter (SPF) arrangement that can be used to process the sparse range elevation azimuth Doppler (READ) cube generated in the radar imaging pipeline shown in FIG. 7A to generate a radar image.

[0015] FIG. 8A schematically illustrates multi-lobe emission from a radar antenna and the reflections generated by a target and clutter in the surrounding environment.

[0016] FIG. 8B schematically illustrates example spectra of the signals reflected from the clutter and the target shown in FIG. 8A.

[0017] FIG. 9A is an image showing a row of vehicles parked along a curb identified by a perception system as depicted by bounding boxes.

[0018] FIG. 9B is a top view two-dimensional (2D) radar image of the scene captured in the image shown in FIG. 9A generated by an existing imaging radar system depicting targets and features identified by the imaging radar system, overlapped with a top view optical image of the scene.

[0019] FIG. 9C is a top view radar image of the scene captured in the image shown in FIG. 9A generated by the disclosed imaging radar systems (e.g., using the radar imaging pipeline shown in FIG. 7A), providing a detailed view of the roadway in addition to the targets.DETAILED DESCRIPTION

[0020] In the following description numerous specific details are set forth in order to provide a thorough understanding of the present disclosure for the purposes of explanation. It will be apparent, however, that the embodiments described by the present disclosure can be practiced without these specific details. In some instances, well-known structures and devices are illustrated in block diagram form in order to avoid unnecessarily obscuring aspects of the present disclosure.

[0021] Specific arrangements or orderings of schematic elements, such as those representing systems, devices, modules, instruction blocks, data elements, and / or theMOTN.153WO PATENTlike are illustrated in the drawings for ease of description. However, it will be understood by those skilled in the art that the specific ordering or arrangement of the schematic elements in the drawings is not meant to imply that a particular order or sequence of processing, or separation of processes, is required unless explicitly described as such. Further, the inclusion of a schematic element in a drawing is not meant to imply that such element is required in all embodiments or that the features represented by such element may not be included in or combined with other elements in some embodiments unless explicitly described as such.

[0022] Further, where connecting elements such as solid or dashed lines or arrows are used in the drawings to illustrate a connection, relationship, or association between or among two or more other schematic elements, the absence of any such connecting elements is not meant to imply that no connection, relationship, or association can exist. In other words, some connections, relationships, or associations between elements are not illustrated in the drawings so as not to obscure the disclosure. In addition, for ease of illustration, a single connecting element can be used to represent multiple connections, relationships or associations between elements. For example, where a connecting element represents communication of signals, data, or instructions (e.g., “software instructions”), it should be understood by those skilled in the art that such element can represent one or multiple signal paths (e.g., a bus), as may be needed, to affect the communication.

[0023] Although the terms first, second, third, and / or the like are used to describe various elements, these elements should not be limited by these terms. The terms first, second, third, and / or the like are used only to distinguish one element from another. For example, a first contact could be termed a second contact and, similarly, a second contact could be termed a first contact without departing from the scope of the described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.

[0024] The terminology used in the description of the various described embodiments herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a,” “an” and “the” are intended to includeMOTN.153WO PATENTthe plural forms as well and can be used interchangeably with “one or more” or “at least one,” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and / or “comprising,” when used in this description specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0025] As used herein, the terms “communication” and “communicate” refer to at least one of the reception, receipt, transmission, transfer, provision, and / or the like of information (or information represented by, for example, data, signals, messages, instructions, commands, and / or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and / or send (e.g., transmit) information to the other unit. This may refer to a direct or indirect connection that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and / or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit (e.g., a third unit located between the first unit and the second unit) processes information received from the first unit and transmits the processed information to the second unit. In some embodiments, a message may refer to a network packet (e.g., a data packet and / or the like) that includes data.

[0026] As used herein, the term “if” is, optionally, construed to mean “when”, “upon”, “in response to determining,” “in response to detecting,” and / or the like, depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining,” “in response to determining,” “upon detecting [the stated condition or event],” “in response to detectingMOTN.153WO PATENT[the stated condition or event],” and / or the like, depending on the context. Also, as used herein, the terms “has”, “have”, “having”, or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise.

[0027] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.General Overview

[0028] In some aspects and / or embodiments, systems, methods, and computer program products described herein include and / or implement improvements to the design, manufacture, operation, and / or management of an autonomous vehicle or fleet of such vehicles, components or elements of an autonomous vehicle or fleet of such vehicles, support systems for an autonomous vehicle or fleet of such vehicles, or ancillary systems related to an autonomous vehicle or fleet of such vehicles.

[0029] By virtue of the implementation of systems, methods, and computer program products described herein, techniques for improving autonomous vehicles or fleets of such vehicles are realized (including improvement to the design, manufacture, operation, and / or management of an autonomous vehicle or fleet of such vehicles, components or elements of an autonomous vehicle or fleet of such vehicles, support systems for an autonomous vehicle or fleet of such vehicles, or ancillary systems related to an autonomous vehicle or fleet of such vehicles).

[0030] Further details regarding particular improvements contemplated herein are provided in sections below. It should be understood that statements made within sections below in the context of wide-angle imaging systems should be understood to refer to the embodiments disclosed in the context of wide-angle imaging systems, and notMOTN.153WO PATENTnecessarily to all embodiments of the present disclosure. For example, statements within in the context of wide-angle imaging systems indicating that certain features are necessary or essential, to the extent such statements exist, should be understood to refer to the necessity or essentiality of such features with respect to the embodiments of in the sections directed to wide-angle imaging systems. Such statements do not indicate that such features are necessary or essential for all embodiments of the present disclosure. Similarly, term definitions in sections below directed to wide-angle imaging systems should be understood to define terms in the context of wide-angle imaging systems, and such definitions may not apply to the remainder of this disclosure.

[0031] The improvements discussed in the context of wide-angle imaging systems may represent computer-implementable improvements implemented within the systems described herein with respect to FIGS. 1-4. For example, the improvements discussed within the sections directed to wide-angle imaging systems may be implemented in whole or in part by a device 300 of FIG. 3, such as by execution on the processor 304 of FIG. 3 of computer code stored within memory 306 of FIG. 3. In one embodiment, the device 300 is located within a vehicle, such as the vehicle 200 of FIG. 2. In another embodiment, the device 300 is located externally to a vehicle, such as in a remote AV system 114, fleet management system 116, or vehicle-to-infrastructure system 118 of FIG. 1. Additionally or alternatively, the improvements discussed within sections directed to wide-angle imaging systems may represent improvement components for elements of FIG. 1, including improvement components of a vehicle 102, a remote AV system 114, fleet management system 116, or vehicle-to-infrastructure system 118 of FIG. 1. Accordingly, the improvements discussed in the context of wide-angle imaging systems may be bodily incorporated into one or more of the above-mentioned elements of FIG. 1 to improvement such elements. As one example, improvements to LiDAR sensors may be used to modify LiDAR sensor 202b of FIG. 2. Further details regarding an example implementation of the LiDAR sensor 202b of FIG. 2 are provided in U.S. Patent Application No. 17 / 931,051, entitled “SYSTEMS AND METHODS FOR TIME-OF-FLIGHT (TOF) LIDAR SIGNAL-TO-NOISE IMPROVEMENT” and filed September 9, 2022 (Att’y docket no: MOTN.057A), the entirety of which is hereby incorporated by reference herein.MOTN.153WO PATENT

[0032] Referring now to FIG. 1, illustrated is an example environment 100 in which vehicles that include autonomous systems, as well as vehicles that do not, are operated. As illustrated, environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, area 108, vehicle-to-infrastructure (V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, fleet management system 116, and V2I system 118. Vehicles 102a-102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, fleet management system 116, and V2I system 118 interconnect (e.g., establish a connection to communicate and / or the like) via wired connections, wireless connections, or a combination of wired or wireless connections. In some embodiments, objects 104a-104n interconnect with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, fleet management system 116, and V2I system 118 via wired connections, wireless connections, or a combination of wired or wireless connections.

[0033] Vehicles 102a-102n (referred to individually as vehicle 102 and collectively as vehicles 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicles 102 are configured to be in communication with V2I device 110, remote AV system 114, fleet management system 116, and / or V2I system 118 via network 112. In some embodiments, vehicles 102 include cars, buses, trucks, trains, and / or the like. In some embodiments, vehicles 102 are the same as, or similar to, vehicles 200, described herein (see FIG. 2). In some embodiments, a vehicle 200 of a set of vehicles 200 is associated with an autonomous fleet manager. In some embodiments, vehicles 102 travel along respective routes 106a-106n (referred to individually as route 106 and collectively as routes 106), as described herein. In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0034] Objects 104a-104n (referred to individually as object 104 and collectively as objects 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, at least one structure (e.g., a building, a sign, a fire hydrant, etc.), and / or the like. Each object 104 is stationary (e.g., located at a fixed location for a period of time) or mobile (e.g., having a velocity and associated with at least one trajectory). In some embodiments, objects 104 are associated with corresponding locations in area 108.MOTN.153WO PATENT

[0035] Routes 106a-106n (referred to individually as route 106 and collectively as routes 106) are each associated with (e.g., prescribe) a sequence of actions (also known as a trajectory) connecting states along which an AV can navigate. Each route 106 starts at an initial state (e.g., a state that corresponds to a first spatiotemporal location, velocity, and / or the like) and ends at a final goal state (e.g., a state that corresponds to a second spatiotemporal location that is different from the first spatiotemporal location) or goal region (e.g. a subspace of acceptable states (e.g., terminal states)). In some embodiments, the first state includes a location at which an individual or individuals are to be picked-up by the AV and the second state or region includes a location or locations at which the individual or individuals picked-up by the AV are to be dropped-off. In some embodiments, routes 106 include a plurality of acceptable state sequences (e.g., a plurality of spatiotemporal location sequences), the plurality of state sequences associated with (e.g., defining) a plurality of trajectories. In an example, routes 106 include only high level actions or imprecise state locations, such as a series of connected roads dictating turning directions at roadway intersections. Additionally, or alternatively, routes 106 may include more precise actions or states such as, for example, specific target lanes or precise locations within the lane areas and targeted speed at those positions. In an example, routes 106 include a plurality of precise state sequences along the at least one high level action sequence with a limited lookahead horizon to reach intermediate goals, where the combination of successive iterations of limited horizon state sequences cumulatively correspond to a plurality of trajectories that collectively form the high level route to terminate at the final goal state or region.

[0036] Area 108 includes a physical area (e.g., a geographic region) within which vehicles 102 can navigate. In an example, area 108 includes at least one state (e.g., a country, a province, an individual state of a plurality of states included in a country, etc.), at least one portion of a state, at least one city, at least one portion of a city, etc. In some embodiments, area 108 includes at least one named thoroughfare (referred to herein as a “road”) such as a highway, an interstate highway, a parkway, a city street, etc. Additionally, or alternatively, in some examples area 108 includes at least one unnamed road such as a driveway, a section of a parking lot, a section of a vacant and / or undeveloped lot, a dirt path, etc. In some embodiments, a road includes at least one laneMOTN.153WO PATENT(e.g., a portion of the road that can be traversed by vehicles 102). In an example, a road includes at least one lane associated with (e.g., identified based on) at least one lane marking.

[0037] Vehicle-to-lnfrastructure (V2I) device 110 (sometimes referred to as a Vehicle-to-Infrastructure or Vehicle-to-Everything (V2X) device) includes at least one device configured to be in communication with vehicles 102 and / or V2I infrastructure system 118. In some embodiments, V2I device 110 is configured to be in communication with vehicles 102, remote AV system 114, fleet management system 116, and / or V2I system 118 via network 112. In some embodiments, V2I device 110 includes a radio frequency identification (RFID) device, signage, cameras (e.g., two-dimensional (2D) and / or three-dimensional (3D) cameras), lane markers, streetlights, parking meters, etc. In some embodiments, V2I device 110 is configured to communicate directly with vehicles 102. Additionally, or alternatively, in some embodiments V2I device 110 is configured to communicate with vehicles 102, remote AV system 114, and / or fleet management system 116 via V2I system 118. In some embodiments, V2I device 110 is configured to communicate with V2I system 118 via network 112.

[0038] Network 112 includes one or more wired and / or wireless networks. In an example, network 112 includes a cellular network (e.g., a long term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN), a private network, an ad hoc network, an intranet, the Internet, a fiber opticbased network, a cloud computing network, etc., a combination of some or all of these networks, and / or the like.

[0039] Remote AV system 114 includes at least one device configured to be in communication with vehicles 102, V2I device 110, network 112, fleet management system 116, and / or V2I system 118 via network 112. In an example, remote AV system 114 includes a server, a group of servers, and / or other like devices. In some embodiments, remote AV system 114 is co-located with the fleet management system 116. In some embodiments, remote AV system 114 is involved in the installation of someMOTN.153WO PATENTor all of the components of a vehicle, including an autonomous system, an autonomous vehicle compute, software implemented by an autonomous vehicle compute, and / or the like. In some embodiments, remote AV system 114 maintains (e.g., updates and / or replaces) such components and / or software during the lifetime of the vehicle.

[0040] Fleet management system 116 includes at least one device configured to be in communication with vehicles 102, V2I device 110, remote AV system 114, and / or V21 infrastructure system 118. In an example, fleet management system 116 includes a server, a group of servers, and / or other like devices. In some embodiments, fleet management system 116 is associated with a ridesharing company (e.g., an organization that controls operation of multiple vehicles (e.g., vehicles that include autonomous systems and / or vehicles that do not include autonomous systems) and / or the like).

[0041] In some embodiments, V2I system 118 includes at least one device configured to be in communication with vehicles 102, V2I device 110, remote AV system 114, and / or fleet management system 116 via network 112. In some examples, V2I system 118 is configured to be in communication with V2I device 110 via a connection different from network 112. In some embodiments, V2I system 118 includes a server, a group of servers, and / or other like devices. In some embodiments, V2I system 118 is associated with a municipality or a private institution (e.g., a private institution that maintains V2I device 110 and / or the like).

[0042] The number and arrangement of elements illustrated in FIG. 1 are provided as an example. There can be additional elements, fewer elements, different elements, and / or differently arranged elements, than those illustrated in FIG. 1. Additionally, or alternatively, at least one element of environment 100 can perform one or more functions described as being performed by at least one different element of FIG. 1. Additionally, or alternatively, at least one set of elements of environment 100 can perform one or more functions described as being performed by at least one different set of elements of environment 100.

[0043] Referring now to FIG. 2, vehicle 200 (which may be the same as, or similar to vehicles 102 of FIG. 1 ) includes or is associated with autonomous system 202, powertrain control system 204, steering control system 206, and brake system 208. In some embodiments, vehicle 200 is the same as or similar to vehicle 102 (see FIG. 1 ). In someMOTN.153WO PATENTembodiments, autonomous system 202 is configured to confer vehicle 200 autonomous driving capability (e.g., implement at least one driving automation or maneuver-based function, feature, device, and / or the like that enable vehicle 200 to be partially or fully operated without human intervention including, without limitation, fully autonomous vehicles (e.g., vehicles that forego reliance on human intervention such as Level 5 ADS-operated vehicles), highly autonomous vehicles (e.g., vehicles that forego reliance on human intervention in certain situations such as Level 4 ADS-operated vehicles), conditional autonomous vehicles (e.g., vehicles that forego reliance on human intervention in limited situations such as Level 3 ADS-operated vehicles) and / or the like. In one embodiment, autonomous system 202 includes operational or tactical functionality required to operate vehicle 200 in on-road traffic and perform part or all of Dynamic Driving Task (DDT) on a sustained basis. In another embodiment, autonomous system 202 includes an Advanced Driver Assistance System (ADAS) that includes driver support features. Autonomous system 202 supports various levels of driving automation, ranging from no driving automation (e.g., Level 0) to full driving automation (e.g., Level 5). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, reference may be made to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, which is incorporated by reference in its entirety. In some embodiments, vehicle 200 is associated with an autonomous fleet manager and / or a ridesharing company.

[0044] Autonomous system 202 includes a sensor suite that includes one or more devices such as cameras 202a, LiDAR sensors 202b, radar sensors 202c, and microphones 202d. In some embodiments, autonomous system 202 can include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), odometry sensors that generate data associated with an indication of a distance that vehicle 200 has traveled, and / or the like). In some embodiments, autonomous system 202 uses the one or more devices included in autonomous system 202 to generate data associated with environment 100, described herein. The data generated by the one or more devices of autonomous system 202 can be used by one or more systems described herein to observe the environment (e.g., environment 100) in which vehicle 200 is located. In some embodiments, autonomous system 202 includesMOTN.153WO PATENTcommunication device 202e, autonomous vehicle compute 202f, drive-by-wire (DBW) system 202h, and safety controller 202g.

[0045] Cameras 202a include at least one device configured to be in communication with communication device 202e, autonomous vehicle compute 202f, and / or safety controller 202g via a bus (e.g., a bus that is the same as or similar to bus 302 of FIG. 3). Cameras 202a include at least one camera (e.g., a digital camera using a light sensor such as a Charge-Coupled Device (CCD), a thermal camera, an infrared (IR) camera, an event camera, and / or the like) to capture images including physical objects (e.g., cars, buses, curbs, people, and / or the like). In some embodiments, camera 202a generates camera data as output. In some examples, camera 202a generates camera data that includes image data associated with an image. In this example, the image data may specify at least one parameter (e.g., image characteristics such as exposure, brightness, etc., an image timestamp, and / or the like) corresponding to the image. In such an example, the image may be in a format (e.g., RAW, JPEG, PNG, and / or the like). In some embodiments, camera 202a includes a plurality of independent cameras configured on (e.g., positioned on) a vehicle to capture images for the purpose of stereopsis (stereo vision). In some examples, camera 202a includes a plurality of cameras that generate image data and transmit the image data to autonomous vehicle compute 202f and / or a fleet management system (e.g., a fleet management system that is the same as or similar to fleet management system 116 of FIG. 1). In such an example, autonomous vehicle compute 202f determines depth to one or more objects in a field of view of at least two cameras of the plurality of cameras based on the image data from the at least two cameras. In some embodiments, cameras 202a is configured to capture images of objects within a distance from cameras 202a (e.g., up to 100 meters, up to a kilometer, and / or the like). Accordingly, cameras 202a include features such as sensors and lenses that are optimized for perceiving objects that are at one or more distances from cameras 202a.

[0046] In an embodiment, camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs and / or other physical objects that provide visual navigation information. In some embodiments, camera 202a generates traffic light data associated with one or more images. In someMOTN.153WO PATENTexamples, camera 202a generates TLD (Traffic Light Detection) data associated with one or more images that include a format (e.g., RAW, JPEG, PNG, and / or the like). In some embodiments, camera 202a that generates TLD data differs from other systems described herein incorporating cameras in that camera 202a can include one or more cameras with a wide field of view (e.g., a wide-angle lens, a fish-eye lens, a lens having a viewing angle of approximately 120 degrees or more, and / or the like) to generate images about as many physical objects as possible.

[0047] Light Detection and Ranging (LiDAR) sensors 202b include at least one device configured to be in communication with communication device 202e, autonomous vehicle compute 202f, and / or safety controller 202g via a bus (e.g., a bus that is the same as or similar to bus 302 of FIG. 3). LiDAR sensors 202b include a system configured to transmit light from a light emitter (e.g., a laser transmitter). Light emitted by LiDAR sensors 202b include light (e.g., infrared light and / or the like) that is outside of the visible spectrum. In some embodiments, during operation, light emitted by LiDAR sensors 202b encounters a physical object (e.g., a vehicle) and is reflected back to LiDAR sensors 202b. In some embodiments, the light emitted by LiDAR sensors 202b does not penetrate the physical objects that the light encounters. LiDAR sensors 202b also include at least one light detector which detects the light that was emitted from the light emitter after the light encounters a physical object. In some embodiments, at least one data processing system associated with LiDAR sensors 202b generates an image (e.g., a point cloud, a combined point cloud, and / or the like) representing the objects included in a field of view of LiDAR sensors 202b. In some examples, the at least one data processing system associated with LiDAR sensor 202b generates an image that represents the boundaries of a physical object, the surfaces (e.g., the topology of the surfaces) of the physical object, and / or the like. In such an example, the image is used to determine the boundaries of physical objects in the field of view of LiDAR sensors 202b.

[0048] Radio Detection and Ranging (radar) sensors 202c include at least one device configured to be in communication with communication device 202e, autonomous vehicle compute 202f, and / or safety controller 202g via a bus (e.g., a bus that is the same as or similar to bus 302 of FIG. 3). Radar sensors 202c include a system configured to transmit radio waves (either pulsed or continuously). The radio waves transmitted by radarMOTN.153WO PATENTsensors 202c include radio waves that are within a predetermined spectrum In some embodiments, during operation, radio waves transmitted by radar sensors 202c encounter a physical object and are reflected back to radar sensors 202c. In some embodiments, the radio waves transmitted by radar sensors 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with radar sensors 202c generates signals representing the objects included in a field of view of radar sensors 202c. For example, the at least one data processing system associated with radar sensor 202c generates an image that represents the boundaries of a physical object, the surfaces (e.g., the topology of the surfaces) of the physical object, and / or the like. In some examples, the image is used to determine the boundaries of physical objects in the field of view of radar sensors 202c.

[0049] Microphones 202d includes at least one device configured to be in communication with communication device 202e, autonomous vehicle compute 202f, and / or safety controller 202g via a bus (e.g., a bus that is the same as or similar to bus 302 of FIG. 3). Microphones 202d include one or more microphones (e.g., array microphones, external microphones, and / or the like) that capture audio signals and generate data associated with (e.g., representing) the audio signals. In some examples, microphones 202d include transducer devices and / or like devices. In some embodiments, one or more systems described herein can receive the data generated by microphones 202d and determine a position of an object relative to vehicle 200 (e.g., a distance and / or the like) based on the audio signals associated with the data.

[0050] Communication device 202e includes at least one device configured to be in communication with cameras 202a, LiDAR sensors 202b, radar sensors 202c, microphones 202d, autonomous vehicle compute 202f, safety controller 202g, and / or DBW (Drive-By-Wire) system 202h. For example, communication device 202e may include a device that is the same as or similar to communication interface 314 of FIG. 3. In some embodiments, communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device that enables wireless communication of data between vehicles).

[0051] Autonomous vehicle compute 202f includes at least one device configured to be in communication with cameras 202a, LiDAR sensors 202b, radar sensors 202c,MOTN.153WO PATENTmicrophones 202d, communication device 202e, safety controller 202g, and / or DBW system 202h. In some examples, autonomous vehicle compute 202f includes a device such as a client device, a mobile device (e.g., a cellular telephone, a tablet, and / or the like), a server (e.g., a computing device including one or more central processing units, graphical processing units, and / or the like), and / or the like. In some embodiments, autonomous vehicle compute 202f is the same as or similar to autonomous vehicle compute 400, described herein. Additionally, or alternatively, in some embodiments autonomous vehicle compute 202f is configured to be in communication with an autonomous vehicle system (e.g., an autonomous vehicle system that is the same as or similar to remote AV system 114 of FIG. 1), a fleet management system (e.g., a fleet management system that is the same as or similar to fleet management system 116 of FIG. 1), a V2I device (e.g., a V2I device that is the same as or similar to V2I device 110 of FIG. 1), and / or a V2I system (e.g., a V2I system that is the same as or similar to V2I system 118 of FIG. 1).

[0052] Safety controller 202g includes at least one device configured to be in communication with cameras 202a, LiDAR sensors 202b, radar sensors 202c, microphones 202d, communication device 202e, autonomous vehicle computer 202f, and / or DBW system 202h. In some examples, safety controller 202g includes one or more controllers (electrical controllers, electromechanical controllers, and / or the like) that are configured to generate and / or transmit control signals to operate one or more devices of vehicle 200 (e.g., powertrain control system 204, steering control system 206, brake system 208, and / or the like). In some embodiments, safety controller 202g is configured to generate control signals that take precedence over (e.g., overrides) control signals generated and / or transmitted by autonomous vehicle compute 202f.

[0053] DBW system 202h includes at least one device configured to be in communication with communication device 202e and / or autonomous vehicle compute 202f. In some examples, DBW system 202h includes one or more controllers (e.g., electrical controllers, electromechanical controllers, and / or the like) that are configured to generate and / or transmit control signals to operate one or more devices of vehicle 200 (e.g., powertrain control system 204, steering control system 206, brake system 208, and / or the like). Additionally, or alternatively, the one or more controllers of DBW system 202h areMOTN.153WO PATENTconfigured to generate and / or transmit control signals to operate at least one different device (e.g., a turn signal, headlights, door locks, windshield wipers, and / or the like) of vehicle 200.

[0054] Powertrain control system 204 includes at least one device configured to be in communication with DBW system 202h. In some examples, powertrain control system 204 includes at least one controller, actuator, and / or the like. In some embodiments, powertrain control system 204 receives control signals from DBW system 202h and powertrain control system 204 causes vehicle 200 to make longitudinal vehicle motion, such as start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate in a direction, decelerate in a direction or to make lateral vehicle motion such as performing a left turn, performing a right turn, and / or the like. In an example, powertrain control system 204 causes the energy (e.g., fuel, electricity, and / or the like) provided to a motor of the vehicle to increase, remain the same, or decrease, thereby causing at least one wheel of vehicle 200 to rotate or not rotate.

[0055] Steering control system 206 includes at least one device configured to rotate one or more wheels of vehicle 200. In some examples, steering control system 206 includes at least one controller, actuator, and / or the like. In some embodiments, steering control system 206 causes the front two wheels and / or the rear two wheels of vehicle 200 to rotate to the left or right to cause vehicle 200 to turn to the left or right. In other words, steering control system 206 causes activities necessary for the regulation of the y-axis component of vehicle motion.

[0056] Brake system 208 includes at least one device configured to actuate one or more brakes to cause vehicle 200 to reduce speed and / or remain stationary. In some examples, brake system 208 includes at least one controller and / or actuator that is configured to cause one or more calipers associated with one or more wheels of vehicle 200 to close on a corresponding rotor of vehicle 200. Additionally, or alternatively, in some examples brake system 208 includes an automatic emergency braking (AEB) system, a regenerative braking system, and / or the like.

[0057] In some embodiments, vehicle 200 includes at least one platform sensor (not explicitly illustrated) that measures or infers properties of a state or a condition of vehicle 200. In some examples, vehicle 200 includes platform sensors such as a globalMOTN.153WO PATENTpositioning system (GPS) receiver, an inertial measurement unit (IMU), a wheel speed sensor, a wheel brake pressure sensor, a wheel torque sensor, an engine torque sensor, a steering angle sensor, and / or the like. Although brake system 208 is illustrated to be located in the near side of vehicle 200 in FIG. 2, brake system 208 may be located anywhere in vehicle 200. like.

[0058] Referring now to FIG. 3, illustrated is a schematic diagram of a device 300. As illustrated, device 300 includes processor 304, memory 306, storage component 308, input interface 310, output interface 312, communication interface 314, and bus 302. In some embodiments, device 300 corresponds to at least one device of vehicles 102 (e.g., at least one device of a system of vehicles 102), at least one device of the remote AV system 114, at least one device of the fleet management system 116, at least one device of the vehicle-to-infrastructure system 118, and / or one or more devices of network 112 (e.g., one or more devices of a system of network 112). In some embodiments, one or more devices of vehicles 102 (e.g., one or more devices of a system of vehicles 102), at least one device of the remote AV system 114, at least one device of the fleet management system 116, at least one device of the vehicle-to-infrastructure system 118, and / or one or more devices of network 112 (e.g., one or more devices of a system of network 112) include at least one device 300 and / or at least one component of device 300. As shown in FIG. 3, device 300 includes bus 302, processor 304, memory 306, storage component 308, input interface 310, output interface 312, and communication interface 314.

[0059] Bus 302 includes a component that permits communication among the components of device 300. In some embodiments, the processor 304 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), and / or the like), a microphone, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), and / or the like) that can be programmed to perform at least one function. Memory 306 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, optical memory, and / or the like) that stores data and / or instructions for use by processor 304.MOTN.153WO PATENT

[0060] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid state disk, and / or the like), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, a CD-ROM, RAM, PROM, EPROM, FLASH-EPROM, NV-RAM, and / or another type of computer readable medium, along with a corresponding drive.

[0061] Input interface 310 includes a component that permits device 300 to receive information, such as via user input (e.g., a touchscreen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, a camera, and / or the like). Additionally or alternatively, in some embodiments input interface 310 includes a sensor that senses information (e.g., a global positioning system (GPS) receiver, an accelerometer, a gyroscope, an actuator, and / or the like). Output interface 312 includes a component that provides output information from device 300 (e.g., a display, a speaker, one or more lightemitting diodes (LEDs), and / or the like).

[0062] In some embodiments, communication interface 314 includes a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, and / or the like) that permits device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. In some examples, communication interface 314 permits device 300 to receive information from another device and / or provide information to another device. In some examples, communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0063] In some embodiments, device 300 performs one or more processes described herein. Device 300 performs these processes based on processor 304 executing software instructions stored by a computer-readable medium, such as memory 305 and / or storage component 308. A computer-readable medium (e.g., a non-transitory computer readable medium) is defined herein as a non-transitory memory device. A non-transitory memory device includes memory space located inside a single physical storage device or memory space spread across multiple physical storage devices.MOTN.153WO PATENT

[0064] In some embodiments, software instructions are read into memory 306 and / or storage component 308 from another computer-readable medium or from another device via communication interface 314. When executed, software instructions stored in memory 306 and / or storage component 308 cause processor 304 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry is used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments described herein are not limited to any specific combination of hardware circuitry and software unless explicitly stated otherwise.

[0065] Memory 306 and / or storage component 308 includes data storage or at least one data structure (e.g., a database and / or the like). Device 300 is capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage or the at least one data structure in memory 306 or storage component 308. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0066] In some embodiments, device 300 is configured to execute software instructions that are either stored in memory 306 and / or in the memory of another device (e.g., another device that is the same as or similar to device 300). As used herein, the term “module” refers to at least one instruction stored in memory 306 and / or in the memory of another device that, when executed by processor 304 and / or by a processor of another device (e.g., another device that is the same as or similar to device 300) cause device 300 (e.g., at least one component of device 300) to perform one or more processes described herein. In some embodiments, a module is implemented in software, firmware, hardware, and / or the like.

[0067] The number and arrangement of components illustrated in FIG. 3 are provided as an example. In some embodiments, device 300 can include additional components, fewer components, different components, or differently arranged components than those illustrated in FIG. 3. Additionally, or alternatively, a set of components (e.g., one or more components) of device 300 can perform one or more functions described as being performed by another component or another set of components of device 300.

[0068] In some embodiments, the device 300 may comprise a high-performance computing (HPC) system of the vehicle 200 configured to perform at least a portion ofMOTN.153WO PATENTprocessing tasks associated with generating an image using data received from one or more of a radar sensor of the radar sensors 202c, a LiDAR sensor of the LiDAR sensors 202b, and a camera of the cameras 202a. In some embodiments, the device 300 may be included in the AV compute 202f.

[0069] In some embodiments, the device 300 may be configured to receive data from a radar sensor, a LiDAR sensor, or a camera via a data link. In some embodiments, the data link can be a wired data link, a wireless data link, or a combination thereof. In some embodiments, where the data link, between a radar sensor, a LiDAR sensor, or a camera, and the device 300 is a wired link, the wired link may comprise a high-speed (e.g., multiGigabit per second) link configured to transmit one or more of video data, radar data, LiDAR data, control data, power, and the like. In some such embodiments, the wired link may comprise a serial link. In some such embodiments, the LiDAR sensor, radar sensor, or camera, and the device 300 each may comprise a serializer / deserializer (SerDes) configured to convert data between serial data carried by the serial link and parallel data transmitted or received by the LiDAR sensor, radar sensor, or camera, and the device 300. In some embodiments, the serial link can be a full-duplex (FX) link. In some embodiments, the data link can be an asymmetric link configured to transport data at a higher rate in a forward direction (e.g., downlink), while simultaneously transporting data at a lower rate in a reverse direction (e.g., uplink). In some embodiments, the wired data link may comprise a coaxial cable or shielded twisted pair cable. In some embodiments, the data link may comprise a gigabit multimedia serial Link (GMSL).

[0070] Referring now to FIG. 4A, illustrated is an example block diagram of an autonomous vehicle compute 400 (sometimes referred to as an “AV stack”). As illustrated, autonomous vehicle compute 400 includes perception system 402 (sometimes referred to as a perception module), planning system 404 (sometimes referred to as a planning module), localization system 406 (sometimes referred to as a localization module), control system 408 (sometimes referred to as a control module), and database 410. In some embodiments, perception system 402, planning system 404, localization system 406, control system 408, and database 410 are included and / or implemented in an autonomous navigation system of a vehicle (e.g., autonomous vehicle compute 202f of vehicle 200). Additionally, or alternatively, in some embodiments perception system 402,MOTN.153WO PATENTplanning system 404, localization system 406, control system 408, and database 410 are included in one or more standalone systems (e.g., one or more systems that are the same as or similar to autonomous vehicle compute 400 and / or the like). In some examples, perception system 402, planning system 404, localization system 406, control system 408, and database 410 are included in one or more standalone systems that are located in a vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in autonomous vehicle compute 400 are implemented in software (e.g., in software instructions stored in memory), computer hardware (e.g., by microprocessors, microcontrollers, application-specific integrated circuits (ASICs), Field Programmable Gate Arrays (FPGAs), and / or the like), or combinations of computer software and computer hardware. It will also be understood that, in some embodiments, autonomous vehicle compute 400 is configured to be in communication with a remote system (e.g., an autonomous vehicle system that is the same as or similar to remote AV system 114, a fleet management system 116 that is the same as or similar to fleet management system 116, a V2I system that is the same as or similar to V2I system 118, and / or the like).

[0071] In some embodiments, perception system 402 receives data associated with at least one physical object (e.g., data that is used by perception system 402 to detect the at least one physical object) in an environment and classifies the at least one physical object. In some examples, perception system 402 receives image data captured by at least one camera (e.g., cameras 202a), the image associated with (e.g., representing) one or more physical objects within a field of view of the at least one camera. In such an example, perception system 402 classifies at least one physical object based on one or more groupings of physical objects (e.g., bicycles, vehicles, traffic signs, pedestrians, and / or the like). In some embodiments, perception system 402 transmits data associated with the classification of the physical objects to planning system 404 based on perception system 402 classifying the physical objects.

[0072] In some embodiments, planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., routes 106) along which a vehicle (e.g., vehicles 102) can travel along toward a destination. In some embodiments, planning system 404 periodically or continuously receives data fromMOTN.153WO PATENTperception system 402 (e.g., data associated with the classification of physical objects, described above) and planning system 404 updates the at least one trajectory or generates at least one different trajectory based on the data generated by perception system 402. In other words, planning system 404 may perform tactical function-related tasks that are required to operate vehicle 102 in on-road traffic. Tactical efforts involve maneuvering the vehicle in traffic during a trip, including but not limited to deciding whether and when to overtake another vehicle, change lanes, or selecting an appropriate speed, acceleration, deacceleration, etc. In some embodiments, planning system 404 receives data associated with an updated position of a vehicle (e.g., vehicles 102) from localization system 406 and planning system 404 updates the at least one trajectory or generates at least one different trajectory based on the data generated by localization system 406.

[0073] In some embodiments, localization system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicles 102) in an area. In some examples, localization system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensors 202b). In certain examples, localization system 406 receives data associated with at least one point cloud from multiple LiDAR sensors and localization system 406 generates a combined point cloud based on each of the point clouds. In these examples, localization system 406 compares the at least one point cloud or the combined point cloud to two-dimensional (2D) and / or a three-dimensional (3D) map of the area stored in database 410. Localization system 406 then determines the position of the vehicle in the area based on localization system 406 comparing the at least one point cloud or the combined point cloud to the map. In some embodiments, the map includes a combined point cloud of the area generated prior to navigation of the vehicle. In some embodiments, maps include, without limitation, high-precision maps of the roadway geometric properties, maps describing road network connectivity properties, maps describing roadway physical properties (such as traffic speed, traffic volume, the number of vehicular and cyclist traffic lanes, lane width, lane traffic directions, or lane marker types and locations, or combinations thereof), and maps describing the spatial locations of road features such as crosswalks, traffic signs or other travel signals of various types. In some embodiments, the map is generated in real-time based on the data received by the perception system.MOTN.153WO PATENT

[0074] In another example, localization system 406 receives Global Navigation Satellite System (GNSS) data generated by a global positioning system (GPS) receiver. In some examples, localization system 406 receives GNSS data associated with the location of the vehicle in the area and localization system 406 determines a latitude and longitude of the vehicle in the area. In such an example, localization system 406 determines the position of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, localization system 406 generates data associated with the position of the vehicle. In some examples, localization system 406 generates data associated with the position of the vehicle based on localization system 406 determining the position of the vehicle. In such an example, the data associated with the position of the vehicle includes data associated with one or more semantic properties corresponding to the position of the vehicle.

[0075] In some embodiments, control system 408 receives data associated with at least one trajectory from planning system 404 and control system 408 controls operation of the vehicle. In some examples, control system 408 receives data associated with at least one trajectory from planning system 404 and control system 408 controls operation of the vehicle by generating and transmitting control signals to cause a powertrain control system (e.g., DBW system 202h, powertrain control system 204, and / or the like), a steering control system (e.g., steering control system 206), and / or a brake system (e.g., brake system 208) to operate. For example, control system 408 is configured to perform operational functions such as a lateral vehicle motion control or a longitudinal vehicle motion control. The lateral vehicle motion control causes activities necessary for the regulation of the y-axis component of vehicle motion. The longitudinal vehicle motion control causes activities necessary for the regulation of the x-axis component of vehicle motion. In an example, where a trajectory includes a left turn, control system 408 transmits a control signal to cause steering control system 206 to adjust a steering angle of vehicle 200, thereby causing vehicle 200 to turn left. Additionally, or alternatively, control system 408 generates and transmits control signals to cause other devices (e.g., headlights, turn signal, door locks, windshield wipers, and / or the like) of vehicle 200 to change states.

[0076] In some embodiments, perception system 402, planning system 404, localization system 406, and / or control system 408 implement at least one machine learning modelMOTN.153WO PATENT(e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, at least one transformer, and / or the like). In some examples, perception system 402, planning system 404, localization system 406, and / or control system 408 implement at least one machine learning model alone or in combination with one or more of the above-noted systems. In some examples, perception system 402, planning system 404, localization system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in an environment and / or the like). An example of an implementation of a machine learning model is included below with respect to FIGS. 4B-4D.

[0077] Database 410 stores data that is transmitted to, received from, and / or updated by perception system 402, planning system 404, localization system 406 and / or control system 408. In some examples, database 410 includes a storage component (e.g., a storage component that is the same as or similar to storage component 308 of FIG. 3) that stores data and / or software related to the operation and uses at least one system of autonomous vehicle compute 400. In some embodiments, database 410 stores data associated with 2D and / or 3D maps of at least one area. In some examples, database 410 stores data associated with 2D and / or 3D maps of a portion of a city, multiple portions of multiple cities, multiple cities, a county, a state, a State (e.g., a country), and / or the like). In such an example, a vehicle (e.g., a vehicle that is the same as or similar to vehicles 102 and / or vehicle 200) can drive along one or more drivable regions (e.g., single-lane roads, multi-lane roads, highways, back roads, off road trails, and / or the like) and cause at least one LiDAR sensor (e.g., a LiDAR sensor that is the same as or similar to LiDAR sensors 202b) to generate data associated with an image representing the objects included in a field of view of the at least one LiDAR sensor.

[0078] In some embodiments, database 410 can be implemented across a plurality of devices. In some examples, database 410 is included in a vehicle (e.g., a vehicle that is the same as or similar to vehicles 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system that is the same as or similar to remote AV system 114, a fleet management system (e.g., a fleet management system that is the same asMOTN.153WO PATENTor similar to fleet management system 116 of FIG. 1, a V2I system (e.g., a V2I system that is the same as or similar to V2I system 118 of FIG. 1 ) and / or the like.

[0079] Referring now to FIG. 4B, illustrated is a diagram of an implementation of a machine learning model. More specifically, illustrated is a diagram of an example implementation of a convolutional neural network (CNN) 420. For purposes of illustration, the following description of CNN 420 will be with respect to an implementation of CNN 420 by perception system 402. However, it will be understood that in some examples CNN 420 (e.g., one or more components of CNN 420) is implemented by other systems different from, or in addition to, perception system 402 such as planning system 404, localization system 406, and / or control system 408. While CNN 420 includes certain features as described herein, these features are provided for the purpose of illustration and are not intended to limit the present disclosure.

[0080] CNN 420 includes a plurality of convolution layers including first convolution layer 422, second convolution layer 424, and convolution layer 426. In some embodiments, CNN 420 includes sub-sampling layer 428 (sometimes referred to as a pooling layer). In some embodiments, sub-sampling layer 428 and / or other subsampling layers have a dimension (i.e., an amount of nodes) that is less than a dimension of an upstream system. By virtue of sub-sampling layer 428 having a dimension that is less than a dimension of an upstream layer, CNN 420 consolidates the amount of data associated with the initial input and / or the output of an upstream layer to thereby decrease the amount of computations necessary for CNN 420 to perform downstream convolution operations. Additionally, or alternatively, by virtue of sub-sampling layer 428 being associated with (e.g., configured to perform) at least one subsampling function (as described below with respect to FIGS. 4C and 4D), CNN 420 consolidates the amount of data associated with the initial input.

[0081] Perception system 402 performs convolution operations based on perception system 402 providing respective inputs and / or outputs associated with each of first convolution layer 422, second convolution layer 424, and convolution layer 426 to generate respective outputs. In some examples, perception system 402 implements CNN 420 based on perception system 402 providing data as input to first convolution layer 422, second convolution layer 424, and convolution layer 426. In such an example, perceptionMOTN.153WO PATENTsystem 402 provides the data as input to first convolution layer 422, second convolution layer 424, and convolution layer 426 based on perception system 402 receiving data from one or more different systems (e.g., one or more systems of a vehicle that is the same as or similar to vehicle 102), a remote AV system that is the same as or similar to remote AV system 114, a fleet management system that is the same as or similar to fleet management system 116, a V2I system that is the same as or similar to V2I system 118, and / or the like). A detailed description of convolution operations is included below with respect to FIG. 4C.

[0082] In some embodiments, perception system 402 provides data associated with an input (referred to as an initial input) to first convolution layer 422 and perception system 402 generates data associated with an output using first convolution layer 422. In some embodiments, perception system 402 provides an output generated by a convolution layer as input to a different convolution layer. For example, perception system 402 provides the output of first convolution layer 422 as input to sub-sampling layer 428, second convolution layer 424, and / or convolution layer 426. In such an example, first convolution layer 422 is referred to as an upstream layer and sub-sampling layer 428, second convolution layer 424, and / or convolution layer 426 are referred to as downstream layers. Similarly, in some embodiments perception system 402 provides the output of sub-sampling layer 428 to second convolution layer 424 and / or convolution layer 426 and, in this example, sub-sampling layer 428 would be referred to as an upstream layer and second convolution layer 424 and / or convolution layer 426 would be referred to as downstream layers.

[0083] In some embodiments, perception system 402 processes the data associated with the input provided to CNN 420 before perception system 402 provides the input to CNN 420. For example, perception system 402 processes the data associated with the input provided to CNN 420 based on perception system 402 normalizing sensor data (e.g., image data, LiDAR data, radar data, and / or the like).

[0084] In some embodiments, CNN 420 generates an output based on perception system 402 performing convolution operations associated with each convolution layer. In some examples, CNN 420 generates an output based on perception system 402 performing convolution operations associated with each convolution layer and an initial input. In someMOTN.153WO PATENTembodiments, perception system 402 generates the output and provides the output as fully connected layer 430. In some examples, perception system 402 provides the output of convolution layer 426 as fully connected layer 430, where fully connected layer 430 includes data associated with a plurality of feature values referred to as F1, F2 . . . FN. In this example, the output of convolution layer 426 includes data associated with a plurality of output feature values that represent a prediction.

[0085] In some embodiments, perception system 402 identifies a prediction from among a plurality of predictions based on perception system 402 identifying a feature value that is associated with the highest likelihood of being the correct prediction from among the plurality of predictions. For example, where fully connected layer 430 includes feature values F1, F2, . . . FN, and F1 is the greatest feature value, perception system 402 identifies the prediction associated with F1 as being the correct prediction from among the plurality of predictions. In some embodiments, perception system 402 trains CNN 420 to generate the prediction. In some examples, perception system 402 trains CNN 420 to generate the prediction based on perception system 402 providing training data associated with the prediction to CNN 420.

[0086] Referring now to FIGS. 4C and 4D, illustrated is a diagram of example operation of CNN 440 by perception system 402. In some embodiments, CNN 440 (e.g., one or more components of CNN 440) is the same as, or similar to, CNN 420 (e.g., one or more components of CNN 420) (see FIG. 4B).

[0087] At step 450, perception system 402 provides data associated with an image as input to CNN 440 (step 450). For example, as illustrated, perception system 402 provides the data associated with the image to CNN 440, where the image is a greyscale image represented as values stored in a two-dimensional (2D) array. In some embodiments, the data associated with the image may include data associated with a color image, the color image represented as values stored in a three-dimensional (3D) array. Additionally, or alternatively, the data associated with the image may include data associated with an infrared image, a radar image, and / or the like.

[0088] At step 455, CNN 440 performs a first convolution function. For example, CNN 440 performs the first convolution function based on CNN 440 providing the values representing the image as input to one or more neurons (not explicitly illustrated) includedMOTN.153WO PATENTin first convolution layer 442. In this example, the values representing the image can correspond to values representing a region of the image (sometimes referred to as a receptive field). In some embodiments, each neuron is associated with a filter (not explicitly illustrated). A filter (sometimes referred to as a kernel) is representable as an array of values that corresponds in size to the values provided as input to the neuron. In one example, a filter may be configured to identify edges (e.g., horizontal lines, vertical lines, straight lines, and / or the like). In successive convolution layers, the filters associated with neurons may be configured to identify successively more complex patterns (e.g., arcs, objects, and / or the like).

[0089] In some embodiments, CNN 440 performs the first convolution function based on CNN 440 multiplying the values provided as input to each of the one or more neurons included in first convolution layer 442 with the values of the filter that corresponds to each of the one or more neurons. For example, CNN 440 can multiply the values provided as input to each of the one or more neurons included in first convolution layer 442 with the values of the filter that corresponds to each of the one or more neurons to generate a single value or an array of values as an output. In some embodiments, the collective output of the neurons of first convolution layer 442 is referred to as a convolved output. In some embodiments, where each neuron has the same filter, the convolved output is referred to as a feature map.

[0090] In some embodiments, CNN 440 provides the outputs of each neuron of first convolutional layer 442 to neurons of a downstream layer. For purposes of clarity, an upstream layer can be a layer that transmits data to a different layer (referred to as a downstream layer). For example, CNN 440 can provide the outputs of each neuron of first convolutional layer 442 to corresponding neurons of a subsampling layer. In an example, CNN 440 provides the outputs of each neuron of first convolutional layer 442 to corresponding neurons of first subsampling layer 444. In some embodiments, CNN 440 adds a bias value to the aggregates of all the values provided to each neuron of the downstream layer. For example, CNN 440 adds a bias value to the aggregates of all the values provided to each neuron of first subsampling layer 444. In such an example, CNN 440 determines a final value to provide to each neuron of first subsampling layer 444MOTN.153WO PATENTbased on the aggregates of all the values provided to each neuron and an activation function associated with each neuron of first subsampling layer 444.

[0091] At step 460, CNN 440 performs a first subsampling function. For example, CNN 440 can perform a first subsampling function based on CNN 440 providing the values output by first convolution layer 442 to corresponding neurons of first subsampling layer 444. In some embodiments, CNN 440 performs the first subsampling function based on an aggregation function. In an example, CNN 440 performs the first subsampling function based on CNN 440 determining the maximum input among the values provided to a given neuron (referred to as a max pooling function). In another example, CNN 440 performs the first subsampling function based on CNN 440 determining the average input among the values provided to a given neuron (referred to as an average pooling function). In some embodiments, CNN 440 generates an output based on CNN 440 providing the values to each neuron of first subsampling layer 444, the output sometimes referred to as a subsampled convolved output.

[0092] At step 465, CNN 440 performs a second convolution function. In some embodiments, CNN 440 performs the second convolution function in a manner similar to how CNN 440 performed the first convolution function, described above. In some embodiments, CNN 440 performs the second convolution function based on CNN 440 providing the values output by first subsampling layer 444 as input to one or more neurons (not explicitly illustrated) included in second convolution layer 446. In some embodiments, each neuron of second convolution layer 446 is associated with a filter, as described above. The filter(s) associated with second convolution layer 446 may be configured to identify more complex patterns than the filter associated with first convolution layer 442, as described above.

[0093] In some embodiments, CNN 440 performs the second convolution function based on CNN 440 multiplying the values provided as input to each of the one or more neurons included in second convolution layer 446 with the values of the filter that corresponds to each of the one or more neurons. For example, CNN 440 can multiply the values provided as input to each of the one or more neurons included in second convolution layer 446 with the values of the filter that corresponds to each of the one or more neurons to generate a single value or an array of values as an output.MOTN.153WO PATENT

[0094] In some embodiments, CNN 440 provides the outputs of each neuron of second convolutional layer 446 to neurons of a downstream layer. For example, CNN 440 can provide the outputs of each neuron of first convolutional layer 442 to corresponding neurons of a subsampling layer. In an example, CNN 440 provides the outputs of each neuron of first convolutional layer 442 to corresponding neurons of second subsampling layer 448. In some embodiments, CNN 440 adds a bias value to the aggregates of all the values provided to each neuron of the downstream layer. For example, CNN 440 adds a bias value to the aggregates of all the values provided to each neuron of second subsampling layer 448. In such an example, CNN 440 determines a final value to provide to each neuron of second subsampling layer 448 based on the aggregates of all the values provided to each neuron and an activation function associated with each neuron of second subsampling layer 448.

[0095] At step 470, CNN 440 performs a second subsampling function. For example, CNN 440 can perform a second subsampling function based on CNN 440 providing the values output by second convolution layer 446 to corresponding neurons of second subsampling layer 448. In some embodiments, CNN 440 performs the second subsampling function based on CNN 440 using an aggregation function. In an example, CNN 440 performs the first subsampling function based on CNN 440 determining the maximum input or an average input among the values provided to a given neuron, as described above. In some embodiments, CNN 440 generates an output based on CNN 440 providing the values to each neuron of second subsampling layer 448.

[0096] At step 475, CNN 440 provides the output of each neuron of second subsampling layer 448 to fully connected layers 449. For example, CNN 440 provides the output of each neuron of second subsampling layer 448 to fully connected layers 449 to cause fully connected layers 449 to generate an output. In some embodiments, fully connected layers 449 are configured to generate an output associated with a prediction (sometimes referred to as a classification). The prediction may include an indication that an object included in the image provided as input to CNN 440 includes an object, a set of objects, and / or the like. In some embodiments, perception system 402 performs one or more operations and / or provides the data associated with the prediction to a different system, described herein.MOTN.153WO PATENT

[0097] As discussed above, the above-noted systems and / or processors may be modified to incorporate the improvements disclosed in the context of wide-angle imaging systems.

[0098] In the foregoing description, aspects and embodiments of the present disclosure have been described with reference to numerous specific details that can vary from implementation to implementation. Accordingly, the description and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. In addition, when we use the term “further comprising,” in the foregoing description or following claims, what follows this phrase can be an additional step or entity, or a sub-step / sub-entity of a previously-recited step or entity.Enhanced Radar Imaging

[0099] As mentioned above, in some embodiments, an autonomous vehicle (AV) such as the vehicle 200 may comprise a sensor suite including different types of sensors including short and long-range LiDARs, mid-range radars, and multiple cameras. These sensors suite, when combined, gives the AV a 360-degree view of the driving environment, allowing it to detect objects as far away as 250 meters, as well as right beside it. This multi-modal approach provides critical redundancy and overlaps across variable driving conditions and scenarios.

[0100] In some embodiments, it can be desirable to reduce the cost and complexity of the sensors in the sensor suite without reducing the functionality of the sensor suite. For example, the performance of one type of sensor having lower cost and complexity may be improved to reduce the number of sensors of another type having higher cost and complexity. For example, by improving the performance of a radar sensor, the number of LiDAR sensors may be reduced. With the advent of advanced and efficient machine learning techniques, high-performance computing systems, and high-speed data links, in some embodiments disclosed herein, the cost and complexity of a sensor system can beMOTN.153WO PATENTreduced by performing some of the data processing tasks, which would be otherwise performed by specialized hardware integrated with a sensor, in the centralized computing system of an AV (e.g., autonomous vehicle compute 202f of the vehicle 200 described above). For example, in some embodiments, processing raw radar data generated by a radar sensor in the centralized computing system of an AV, not only reduces the cost and complexity of the radar sensor but also improves the information content of a radar image generated based on raw radar data.

[0101] LiDAR has long been a cornerstone sensing modality, and in some embodiments, the primary three-dimensional sensor in autonomous vehicles (AVs) enabling safe and comfortable Level 4 self-driving capabilities as defined by the society of automotive engineers (SAE). However, LiDAR remains one of the most expensive and complex components in an AV, even despite per unit costs dropping significantly as the AV industry scales up toward mass commercialization. LiDAR is by far the most expensive sensor on AVs, and the supply chain is still working on producing automotive grade versions.

[0102] Radar imaging presents a low-cost alternative to LiDAR in many applications including autonomous driving, offering robust performance in particular in challenging weather conditions such as fog, heavy rain, or snow where LiDAR can struggle. Unlike LiDAR, which relies on precise laser-based depth perception, radar operates using radio waves that can penetrate obstacles and provide reliable object detection even in low-visibility environments. Radar can be generally more cost-effective and consumes less power, making it a practical choice for widespread adoption in autonomous vehicle systems. While LiDAR excels in high-resolution mapping, the techniques described herein help improve the quality of radar imaging resolution, helping bridge the gap in perception accuracy. Using radar imaging, autonomous vehicles can enhance safety, reduce costs, and achieve better adaptability in real-world driving conditions.

[0103] In some of the embodiments described below, modifications to the radar system architecture, specifically the radar imaging pipeline, have been made by shifting certain signal processing tasks, traditionally handled by the radar sensor hardware, to a computing system (e.g., high-performance computing system of an AV).MOTN.153WO PATENT

[0104] In addition to reducing the complexity of radar sensor hardware, this approach has enabled usage of advanced machine learning techniques to analyze and process certain portions of low-level radar data, which may be discarded in the existing radar systems, to enhance radar performance. Combination, of computational power of a high-performance computing (HPC) system, which allows processing a larger portion of the low-level radar data, and employment of advanced machine learning techniques can improve the images generated by an enhanced radar system to a point that it can rival the point clouds produced by LiDAR. Usage of machine learning methods, enabled by shifting certain signal processing tasks to a high-performance computing system, may enable improved use of radar signals to perceive an environment. This development can open the door to increased reliance on radars as a primary sensing modality in Level 4 AVs, which in turn would reduce vehicle costs while maintaining system safety and performance. By reducing vehicle costs and improving durability without sacrificing safety and ride quality, the radar systems and architecture disclosed here can aid the long-term commercial success of AVs.

[0105] In various embodiments, a radar can generate and receive radio waves having frequencies in high frequency or extremely high frequency (EFH) range (e.g., from 30 GHz to 300 GHz). In some embodiments, a radar can use millimeter waves to detect objects and generate an image of the surrounding environments. Such millimeter wave (mm-Wave) rada can distinguish itself from LiDAR in several crucial capabilities. For example, it can detect objects at distances well beyond 200 meters and reliably identify smaller objects that might otherwise fall between LiDAR scanning lines. Additionally, radar directly measures an object's velocity through Doppler measurement and can maintain its performance even in adverse weather conditions. These latter features are critical for safety and defining the operational design domain (ODD) of an AV system. Radar has also long been used in military, industrial, and commercial applications, and has demonstrated durability.

[0106] Despite all advantages mentioned above, existing radar systems have not been used as primary perception modality in AVs due to certain limitations. One of the main limitations of existing radar systems is generating sparse point clouds that may not allow downstream tasks (specifically perception, prediction, and motion planning) to beMOTN.153WO PATENTperformed with sufficient resolution and accuracy. This is primarily due to constraints imposed by processing bottlenecks in the existing radar systems on the density, size, resolution and accuracy of images and point clouds.Enhanced Radar System

[0107] As mentioned above, part of overcoming the limitation of the current imaging radar systems is modifying the existing imaging radar architecture to eliminate some or all of Digital Signal Processor (DSP) modules used in a radar sensor, by performing the corresponding signal processing tasks in centralized computing system that has significantly more computational power than the DSP modules. In some embodiments, an enhanced radar architecture comprises a centralized low-level radar architecture where low-level radar data generated by reflection of radio waves is directly transmitted to a high-performance computing (HPC) system (e.g., HPC system of an AV) for generating a radar image.

[0108] Existing imaging radar architectures often lose much of the low-level radar data due to bottlenecks in the DSP modules and Ethernet-based datalinks. FIG. 5A is a block diagram showing the building blocks and data flow in a radar system 500. The radar system 500 illustrates some of the inherent limitations in existing imaging radar architecture. Notably, the radar system 500 can be constrained by computing capacities and bandwidth of the hardware embedded in individual radar modules 501a, 501b, leading to severe information bottlenecks along the imaging pipeline resulting in a slow detection rate and limited output. In the example shown, the radar system 500 includes two radar modules 501a, 501b, that are in communication with the AV HPC 510 via ethernet-based data links 506. In some embodiments, the AV HPC 510 can be included in AV compute 202f of the vehicle 200 described above with respect to FIG. 2.

[0109] In some embodiments, a first radar module 501 a may include a first plurality of antennas 508a (e.g., including transmitting and receiving antennas) connected to a first radar radio frontend 502a. The first radar radio frontend 502a can include one or more radio transceivers configured to provide outgoing radio signals to transmit antennas and receive reflected radio signals from the receiving antennas of the first plurality of antennas 508a. The reflected radio signals can be reflections of outgoing radio signals radiated byMOTN.153WO PATENTthe transmitting antenna generated by objects and surface in an environment surrounding the first radar module 501a. In some embodiments, the first radar radio frontend 502a may include an analog-to-digital converter (ADC) that receives analog reflected radio signals and converts them to digitized reflected radio signals. In some embodiments, the digitized reflected radio signals output by a radar radio front end may be referred to as raw radar or low-level data. The first radar radio frontend 502a further includes a first radar DSP module 504a that receives the digitized reflected radio signals via a highspeed link (e.g., 5 Gbps or higher) and processes the digitized reflected radio signals (or raw radar data) to generate processed radar data (e.g., data communicated via the ethemet-based data links 506). In various implementations, the DSP module 504a may filter the raw (or low-level) radar data, perform range processing, Doppler processing, angle finding among other processes to generate processed data. In some embodiments, processed data generated and output by the first DSP module 504a may comprise a range-Doppler cube (RDC). In some examples, RDC can be a 3D data structure that represents radar reflections across range, Doppler frequency, and angle. In some embodiments, the processed data output by the first radar module 501a (e.g., by the first radar DSP module of the first radar module 501a) may be transmitted to the AV HPC 510 via an ethernet link 506. The AV HPC 510 can use the processed data (e.g., an RDC, detections report, or point cloud) to generate a visual representation of the detected objects, e.g., based on their distance and velocity. In some embodiments, the visual representation may be generally referred to as a radar image. In some examples, the first radar module 501 a may generate a few hundred detections per frame, equating to several thousand detections per second. The second radar module 501b may include a second plurality of antennas 508b, a second radar radio frontend 502b, and a second radar DSP 504b. The second radar module 501b may comprise one or more features described above with respect to first radar module 501a and outputs a different processed data generated based on reflected radio signals corresponding to radio waves radiated by the second plurality of the antennas 508b.

[0110] In some embodiments, the first and second radio modules 501 a, 501 b, may be placed at different locations with respect to a vehicle (e.g., an AV) to image different portions of the environment surrounding the vehicle. The first and second pluralities ofMOTN.153WO PATENTantennas may each include different numbers of antennas having different radiation patterns and converting different fields of views. Although the radar system 500 includes two radar modules 501a, 501b, in various implementations, a radar system may include more than two radio sensors positioned at different locations to cover a larger portion of the environment surrounding a vehicle. In some embodiments, a radar sensor may include multiple channels each channel corresponding to one of the antennas of the plurality of antennas.

[0111] The architecture of the radar system 500, which uses a DSP in each radar sensor to process raw radar data prior to transmission to AV HPC 510, imposes a constraint on the amount of raw radar data used to generate the processed data. This is primarily due to computational limitation of the DSP and bandwidth of data links between radar sensor and the AV HPC that result in bottlenecks limiting the density, size, resolution and accuracy of images and point clouds that can be generated using the processed data generated by the radar sensor. For example, due limitations of the computational resources the radar system 500 may prioritize the strongest signals while setting aside semantic information in low-level radar data resulting in loss of potentially relevant information.

[0112] As described above, to address the deficiencies of the existing radar system in particular with respect to constraints associated with limited computational power available in the radar sensor, in the enhanced radar systems and architectures described below, the raw radar data generated by radar radio frontend of an individual radio sensor is directly transmitted to the AV HPC via a high-speed data link (e.g., a serial data link) such that the computational power of the HPC can be used not only to retain and process a larger portion of raw radar data generated by a radio frontend, but also use advanced processing methods and machine learning algorithms in various steps of processing (e.g., range processing, Doppler processing, and angle finding). For example, processing the raw radar data using the HPC may allow spatially varying range processing and frequency varying Doppler processing in contrast to fixed range processing and fixed-frequency Doppler processing used in existing radar system like the radar system 500.

[0113] FIG. 5B is a block diagram illustrating example building blocks and dataflow in an enhanced radar system 520 according to some embodiments disclosed herein. TheMOTN.153WO PATENTenhanced radar system 520 can be an imaging radar system configured to generate an image of the environment surrounding a vehicle using one or more radar units 511 a, 511 b (generically referred to as radar unit 511).

[0114] In some embodiments, unlike the existing imaging radar systems, the radar system 520 uses data processing in a HPC system and the low-level radar data in tandem to generate high-fidelity radar imagery. In some implementations, radar images generated by the enhanced radar system 520 may achieve imaging performance comparable to a LIDAR system generating over 2 million points per second.

[0115] In some embodiments, the radar system 520 may comprise multiple radar sensors, each in communication with the AV HPC 510 via a separate data link. In the example shown, the radar system 520 comprises first and second radar units 511 a, 511 b, each configured to generate low-level radar data and transmit the low-level radar data to the AV HPC 510 for generation a radar image corresponding to a portion of environment electromagnetically interrogated by the respective radar sensor, however, it will be understood that the radar system 520 may include fewer or more radar units 511. In some embodiment first and second radar units 511a, 511b, may be simplified radar modules that directly transmit raw or low-level data, comprising digitized data obtained by digitizing the reflected signals received from the corresponding transceivers, to the AV HPC 510. In some such cases, the radar units 511 may omit (or not include) a radar DSP 504 or bypass a radar DSP 504 and communicate the raw or low-level data to the AV HPC 510.

[0116] In some embodiments, the first radar unit 511a includes a first radar radio frontend 502a and a first plurality of antennas 508a connected to the first radar radio frontend 502a. In some embodiments, the first radio frontend 502a may include a transceiver configured to transmit probe signals and receive reflected signals to and from the first plurality of antennas 508a. In some examples, the first plurality of antennas 508a may include transmit antennas configured to receive the probe signals and emit electromagnetic waves within a first field of view of first radar unit 511 a and receive antennas configured to receive the reflections of the emitted electromagnetic waves and generate the reflected signals. In some embodiments, the first radar radio frontend 502a may include a first ADC configured to digitize the reflected signals to generate the low-level radar data. In some embodiments, the first radar unit 511a includes a first radarMOTN.153WO PATENTradio frontend 502a and a first plurality of antennas 508a connected to the first radar radio frontend 502a. In some embodiments, the second radar radio frontend 502b may comprise one or more features described above with respect to the first radar radio frontend 502a.

[0117] In some embodiments, the first radar radio frontend 502a can be connected to the AV HPC 510 by a first high-speed data link 512a, and the second radar radio frontend 502b is connected to the AV HPC 510 by a second high-speed data link 512b. In some embodiments, one or both the first and second data links 512a, 512b, can be a highspeed (e.g., multi-Gigabit per second) link configured to transmit low-level radar data generated by the first and second radar radio frontends 502a, 502b, to the AV HPC 510. In some embodiments, low-level radar data may comprise digitized radar data generated by an ADC in a corresponding radar radio frontend. In some embodiments, one or both the first and second data links 512a, 512b, may be further configured to transmit control data and power. In some embodiments, one or both the first and second data links 512a, 512b can be a serial link (e.g., an asymmetric data link). In some embodiments, one or both the first and second data links can be full-duplex (FX) links. In some embodiments, the first and second data links 512a, 512b, can be established by two separate coaxial cables or shielded twisted pair cables.

[0118] In some embodiments, one or both the first and second radar radio frontends 502a, 502b, and the AV HPC 510 can include a serializer / deserializer (SerDes) configured to convert data between serial data carried by the first and second data links 512a, 512b and parallel data transmitted by the first and second radar radio frontends 502a, 502b, or received by the AV HPC 510. For example, the first and second radar radio frontends 502a, 502b, may each include a serializer that serializes digitized low-level radar data prior to transmission via the first and second data links 512a, 512b. In some embodiments, the radar system 520 may include a deserializer configured to deserialize serialized data received from the one or both first and second radio frontends 502a, 502b, and to retrieve digitized low-level radar data. In some embodiments, the deserializer may be implemented in the AV HPC 510 as machine-readable instructions stored in a non-transitory memory and by a processor of the AV HPC 510, respectively. In some embodiments, the deserializer may comprise an electronic circuit separate fromMOTN.153WO PATENTthe AV HPC 510 and connected between one or both first and second radio frontend 502a, 502b, and the AV HPC 510.

[0119] In some embodiments, one or both the first and second data links 512a, 512b, can be gigabit multimedia serial links (GMSL).

[0120] In some embodiments, the architecture of the radar system 520 significantly enhances the imaging radar system's capability to interpret complex environments and facilitates a comprehensive understanding of the surroundings environment (e.g., an environment surrounding a vehicle) with minimal latency. In some embodiments, by removing the information bottlenecks (e.g., the radar DSP 504 or use of the radar DSP 504), the radar system 520 can allow the HPC system (e.g., hardware and software stack of an autonomous vehicle (AV)), to fully leverage nearly all information embedded in collected the low-level reflection data. In some embodiments, the radar system 520 can use constellation of radar frontends (e.g., millimeter wave radar frontends) to generate reflection data at multi-Gbps rates, capturing a wealth of information. Processing the low-level reflection data directly by the high-performance computing (HPC) system can allow extraction of information that would be lost in the existing radar imaging pipelines. In some embodiments, this additional information can enhance the radar's capability to contribute to the overall functionality and safety of AVs.

[0121] FIG. 6 is a block diagram illustrating an enhanced radar system 600 comprising multiple radar units 602-1, ..., 602-N (generically referred to as radar unit 602) each configured to generate low-level data carrying all or nearly all information captured by the reflected electromagnetic waves received by the respective radar sensors. In some embodiments, the enhanced radar system 600 may comprise one or more features described above with respect to the radar system 502 described above with respect to FIG. 5B and / or can be an example implementation of the radar system 502.

[0122] In some embodiments, the enhanced radar systems 600 and 502 can be imaging radar systems configured to generate visual representations (radar images) of objects, terrain, or environments using radio waves. In some examples, a radar image can include detailed spatial information, often in the form of 2D or 3D images, and velocities of objects captured by the radar image.MOTN.153WO PATENT

[0123] In some implementations, the enhanced radar systems 600 and 502 can be imaging can be millimeter-wave radar systems used in an automotive application for high resolution imaging on an environment surrounding a vehicle (e.g., and autonomous vehicle). In some implementations, the perception and / or control system of the vehicle may use the radar images to control the vehicle and prevent collisions with objects and obstacles in the environment.

[0124] In some embodiments, individual one of the radar units 602-1, ..., 602-N, can be radar sensors comprising the features described above with respect to the first radar unit 511a described above.

[0125] In some embodiments, the enhanced radar system 600 can be radar system of a vehicle (e.g., an AV) that includes the AV HPC 510 configured to perform the computing tasks associated with processing data received from the radar sensor and, in some embodiments, other sensors (e.g., audio, video, LiDAR, and like) of the AV, to perceive an environment surrounding the vehicle and the objects within the environment. The enhanced radar system 600 includes a radar data processing system 606 implemented in the AV HPC 510 and configured to receive low-level radar data streams 604-1 , ... , 604-N from the radar sensors, and generate a radar image 609, which may be used for perception and object detection. In some embodiments, the radar data processing system 606 may be configured to process the low-level data streams 604-1,..., 604-N to extract, e.g., based on a reflected signal, spatial dimensions (e.g., three dimensions), range (or distance) to objects, azimuth (horizontal angle), elevation (vertical angle), velocity (e.g., radial velocity, or how fast a point is moving toward or away from a respective radar sensor). In some embodiments, azimuth and elevation can be extracted by processing the low-level data in the spatial and / or frequency domain, and the range and the velocity may be extracted by processing the low-level data in the frequency domain. In some embodiments, the radar image 609 may include a radar point cloud (e.g., a four dimensional or4D radar point cloud) associated with a portion of environment surrounding the vehicle and interrogated by the plurality of radar units 602-1, ..., 602-N, using radio waves. In some embodiments, the radar point cloud may include one or more of a three-dimensional (3D) coordinate, a velocity vector, a signal strength (or reflectivity), and a time stamp for an individual point.MOTN.153WO PATENT

[0126] As described above, the radar system 600 can be significantly different from existing radar systems that typically include DSP pipelines in the individual radar sensors and leading to information loss prior to data processing in the HPC due to slow or limited DSP capabilities. In some embodiments, by leveraging the power of the HPC, the radar system 600 can use machine learning modules to process a larger portion of the low-level data received from the radar units 602-1...602-N, more effectively and thereby generate a radar image containing more information in a comparable, and in some embodiments, shorter time period. In some embodiments, the radar data processing system 606 may aggregate all or nearly all low-level radar data in a multi-channel and multi-scan (MCMS) configuration. This technique can produce high-fidelity, low-latency radar images at a rate ranging from 10 to 20 Hz, 20 to 30 Hz, 30 to 40 Hz, or larger, achieving a level of detail comparable to a LiDAR system generating 2 million points per second.

[0127] In some embodiments, a low-level radar data stream 604-i (where, i=1 ...N) generated by the radar unit 602-i can be received by a multi-channel multi-scan (MCMS) processing module 606-i configured to process low-level radar data generated by multiple channels of the radar unit 602-i as a result of multiple scans. In some embodiments, the low-level data stream 604-i can be digitized radar data that have been serialized for transmission over a high-speed serial data link. In some examples, the high-speed data link can be a GMSL such as GMSL1 , GMSL2, or GMSL3. In various implementations, the high-speed data link can transmit the radar data (e.g., serialized and digitized radar data) at a rate ranging from 1 to 2 (Gigabit per second) Gbps, 2 to 3 Gbps, from 3 to 3.5 Gbps, from 3.5 to 4 Gbps, from 4 to 6 Gbps, from 6 to 8 Gbps, or any ranges formed by these values or higher rates. In some embodiments, each low-level radar data stream may correspond to a channel of the radar unit 602-i and can be generated using one of the antennas of the plurality of antennas in the radar unit 602-i. In some embodiments, an individual scan may correspond to a specific radiation pattern generated by one or more of the antennas of the plurality of antennas in the radar unit 602-i. In some embodiments, an individual scan can be performed with a particular set of radar waveform parameters to obtain an observation or sample of a target area independent of preceding or succeeding scans (also referred to as a “look”). In some examples, generating an imageMOTN.153WO PATENTusing multiple scans (or looks) can resolve ambiguities in range, azimuth, elevation, or velocity and average out random variations from noise, leading to a clearer and more reliable radar image.

[0128] In some embodiments, the radar unit 602-i can operate at F Hertz generating F frames of data in one second. In various implementations, F can be from 1 to 3 (Gigabit per second) Gbps, 3 to 4 Gbps, 4 to 6 Gbps, 6 to 8 Gbps, or any ranges formed by these values or higher rates

[0129] In some such embodiments, to generate an individual frame, the radar radio front end module of the radar unit 602-i transmits a waveform with a certain set of parameters and the waveform may change for different frames (different scans). Effectively the signal and / or radiation pattern transmitted by the antenna can change for capturing each frame. The multi-channel multi-scan (MCMS) processing module 606-i can combine low-level radar data across successive scans to generate processed radar data 614-i usable for generating a radar image of a target scene of the radar unit 602-i. In some embodiments, individual scans can be configured such the multi-channel multiscan (MCMS) processing module 606-i can generate processed radar data 614-i having more information or more accurate information about the target scene compared to processed data generated using a single scan.

[0130] In some embodiments, processed radar data 614-i generated by an individual MCMS processing module 606-i can include information about multiple points in the target scene of the radar unit 602-i. The information can include position in three-dimensional (3D) space, a velocity vector, a signal strength (or reflectivity), and / or a time stamp. In some embodiments, the processed radar data 614-i generated by an individual MCMS processing module 606-i may comprise a radar image. However, the embodiments are not so limited, and the information can include other data.

[0131] In some embodiments, the radar system 600 may include a MCMS aggregation module 608 that aggregates processed radar data generated by individual MCMS processing modules 606-1 , ... , 606-N, to generate the radar image 609. The radar image 609 may comprise a range-Doppler map visually representing detected objects in an environment based on their distance and velocity and at least a portion of the background and clutter in the environment.MOTN.153WO PATENT

[0132] In some embodiments, the processed radar data 614-i output by an individual MCMS processing (or imaging) module 606-i can be an RDC. In some other embodiments, the MCMS aggregation module 608 may use processed radar data received from two or more of the MCMS processing (or imaging) modules 606-1 , ... ,606-N to generate an RDC that includes information from the corresponding low-level radar data streams and process the RDC to generate the radar image 609.

[0133] In some cases, the output of an individual MCSM imaging module can be a radar image and the MCMS aggregation module 608 may combine multiple radar images to generate a single radar image 609. For example, a first MCSM imaging module may generate a first radar image of a first scene in the environment captured by a first radar unit, a second MCSM imaging module may generate a second radar image of a second scene in the environment captured by a second radar unit, and the MCMS aggregation module 608 may use the first and second radar images to generate a composite radar image 609 comprising the first and the second image or a radar image comprising pixels / points corresponding to the first and second scene.

[0134] In some embodiments, the radar image 609 may be provided to an object detection module 610 implemented in the AV HPC 510 to identify objects in the radar image 609 and generate a tagged radar map 612 comprising the radar image 609 and additionally markers and identifiers indicative of different objects in the image and the spatial coordinate, range, and velocity for individual objects.

[0135] In some embodiments, the MCMS processing (or imaging) modules 606-1 ...606-N, the MCMS aggregation module 608, the MCMS aggregation module 608, and the object detection module 610 may be implemented in the AV HPC 510 as machine readable data and instructions stored in a non-transitory memory of the AV HPC 510 and executed by a hardware processor of the AV HPC 510.

[0136] In some embodiments, at least a portion of an individual MCMS processing module 606-i may be implemented using machine learning techniques and training data collected and processed by various test systems. In various implementations, the machine learning techniques may be used for determining window functions in spatial and frequency domains, dynamically adjusting parameters of a range or Doppler processor, and / or other processing and computational tasks.MOTN.153WO PATENT

[0137] In some embodiments, the radar units 511a / b of the enhanced radar system 520 and the radar units 602-1 / 2 of the enhanced radar system 600 may be separated from AV HPC 510. In some embodiments, individual radar units 511a / b of the enhanced radar system 520 and individual radar units 602-1 / 2 of the enhanced radar system 600 may be positioned at different locations (e.g., different locations of a vehicle). For example, the radar unit 602-1 may be positioned near a front end of the vehicle (e.g., to monitor a first target scene comprising a portion of the environment in front of the vehicle), and the radar unit 602-N may be positioned near a backend of the vehicle (e.g., to monitor a second target scene comprising a portion of the environment behind the vehicle).

[0138] In some embodiments, the radar units 511a / b of the enhanced radar system 520 and the radar units 602-1 / 2 of the enhanced radar system 600 may be connected to the AV HPC 510 via a wired link, a wireless link, or a combination thereof, to transmit low-level (raw) radar data to the AV HPC 510. In some embodiments, a wired link that communicatively connects the AV HPC 510 to a radar unit (e.g., one of the radar units 511a / b or radar units 602-1 / 2) can be an off-chip or off-board data link (e.g., a cable).

[0139] In some examples, the wired link may comprise a coaxial cable or an ethernet cable. In some embodiments, low-level (raw) radar data may comprise reflected radar signals generated by a transceiver of the corresponding radar unit. For example, the low-level (raw) radar data may comprise digitized data output by an analog-to-digital converter of the corresponding radar unit in response to receiving reflected radar signals from the transceiver. In some embodiments, the radar units 511a / b of the enhanced radar system 520 and the radar units 602-1 / 2 of the enhanced radar system 600 may not include a digital signal processing (DSP) circuit (e.g., a DSP circuit configured to process low-level or raw data generated by a transceiver or an A-to-D directly connected to a transceiver). In various embodiments, an individual radar unit 511a / b of the enhanced radar system 520 or an individual units 602-1 / 2 of the enhanced radar system 600 may include a digital signal processing (DSP) circuit that is not used to process low-level or raw data generated by a corresponding transceiver or an A-to-D directly connected to the transceiver.

[0140] In some embodiments, an individual radar unit (e.g., one of the radar units 511a / b or radar units 602-1 / 2) may be included in an enclosure separate for the AV HPC 510. For example, the radar units 511a / b or radar units 602-1 / 2 may be located at theMOTN.153WO PATENTfront or back of a vehicle and the AV HPC 510 may be located in a trunk or other location of the vehicle.

[0141] In some embodiments, an individual radar unit (e.g., one of the radar units 511a / b or radar units 602-1 / 2) may be fabricated on a board or substrate separate for an electronic board or substrate on which the AV HPC 510 is formed.

[0142] In some embodiments, the electronic circuitry of an individual radar unit (e.g., one of the radar units 511a / b or radar units 602-1 / 2) of an enhanced radar system (e.g., radar system 520 or radar system 600), may be configured receive signals from one or more antennas, amplify / condition the signals, and digitize the amplified / conditioned signals for transmission to which the AV HPC 510 is via an off-chip data link.Radar data processing pipeline

[0143] FIG. 7A schematically illustrates an example radar data processing pipeline 700 that may be used in the imaging radar systems shown in FIGS. 5B and 6 to generate a radar image from low-level data streams received from individual radar sensors, according to certain embodiments disclosed herein. In some examples the radar data processing pipeline 700 can be a radar imaging pipeline or a perception radar imaging pipeline.

[0144] In some embodiments, the radar data processing pipeline 700 may be implemented in the AV HPC 510 as machine readable data and instructions stored in a non-transitory memory of the AV HPC 510 and executed by a hardware processor of the AV HPC 510. In some embodiments, radar data processing pipeline 700 can be a data processing pipeline in an individual MCMS processing module 606-i and configured to receive the low-level radar data 701 (e.g., digitalized low-level radar data) and process the low-level radar data 701 to generate processed radar data 614 (e.g., radar image).

[0145] In some embodiments, the low-level radar data 701 may be generated by the radar data processing system 606 by deserializing low-level radar data stream 604-i received from a corresponding radar unit 602-i.

[0146] In the illustrated example of figure 7A, the radar data processing pipeline 700 is configured to generate processed radar data 614 (e.g., a radar image) from digitized low-level radar data 701 (also referred to herein as ADC samples 701). In someMOTN.153WO PATENTembodiments, the digitized low-level radar data 701 may comprise digitized radar signals generated by transceiver of an individual radar unit 602-i in response to receiving radio waves reflected by a target scene of the radar unit 602-i.

[0147] In some embodiments, a portion of reflected radio waves received by the antennas of a radar sensor may be reflected from background or objects other than target objects or objects of primary interest in a target scene of the environment. In some embodiments, background comprises at least one background object. In some such cases, the background object can be a static object (e.g., a building, a road surface, a curb, a post, or a tree).

[0148] In some embodiments, reflected radar signals and low-level radar data generated by this portion of reflected radio waves may be referred to as clutter. For example, in an urban environment, the target in a radio signal may correspond to a vehicle and clutter in the radio signal may correspond to buildings, road surface, curbs, trees, etc. Similar to a target signal, a clutter signal may include a main lobe (corresponding to a particular piece of clutter) and multiple sidelobes (corresponding to ripples of the clutter). The target objects, objects other than the target objects, and the background in an environment may be collectively referred to as radar-scatterers in the environment.

[0149] While an existing radar system (e.g., the radar system 500) may remove clutter and focus solely on a target (e.g., main lobe of a reflected radar signal associated with the target), the HPC, using the perception radar data processing pipeline 700, can additionally retain clutter information (e.g., main lobe of a reflected radar signal associated with the clutter) to improve the resolution of the resulting radar image 609 and facilitate understanding of the environment by a perception system. In some embodiments, the HPC may configure the radar data processing pipeline 700 by relieving certain constraints on spatial filtering, temporal filtering, and adaptive thresholding typically used in existing radar systems to distinguish between target signals and clutter.

[0150] In some embodiments, the HPC may use a machine learning model to retain clutter information when processing low-level radar data to generate processed radar data comprising one or more of a range, velocity, elevation, and azimuth of target objects in the environment.MOTN.153WO PATENT

[0151] In some embodiments, an environment monitored by a radar unit 602 of the radar system 600 may include at least one target object and at least one background (e.g., a background object), and the radar data generated by the radar unit may comprise target data generated by the target object and clutter data generated by the background. In some such embodiments, the AV HPC 510 may retain at least a portion of clutter data when processing the target data and generate the processed data using both target data and clutter data. In some such embodiments, the HPC may retain at least a portion of clutter data when processing the target data at least in part by varying the at least one processing parameter (e.g. based at least in part by using a machine learning algorithm). In some examples, the AV HPC 510 may be configured to use the processed radar data to generate a radar image comprising the target object and the background. In some embodiments, the processing parameter may comprise a type of a window function, a parameter of window function (e.g., a length, a shape, a slope, or the like), a bin size, or another parameter.

[0152] The example radar data processing pipeline 700 includes a spatial varying range processing block 702, frequency varying Doppler processing block 704, first semantic preserving filter 706, MIMO processing block 708, super-resolution angle finding block 710, and second semantic preserving filter 712. The various processing blocks 702-712 may be implemented using the AV HPC 510 implemented using one or more CPUs or GPUs and non-transitory memories storing machine readable instructions, learning data, reference data and other data usable for radar data processing using signal processing techniques and machine learning techniques. In the example shown, different blocks of the radar data processing pipeline 700 are serially connected such that the radar data is sequentially processed to transform the digitized low-level radar data 701 to the processed radar data 614. However, embodiments are not so limited, and, in some embodiments, two or more blocks may be connected in parallel to process common or different inputs during overlapping periods. At the spatial varying range processing block 702, the HPC may process digitized low-level radar data 701 to determine distance of an object or point in the environment (e.g., a target scene of the corresponding radar sensor from which the digitized low-level radar data 701 is received) from the radar system (e.g., the corresponding radar sensor).MOTN.153WO PATENT

[0153] In some embodiments, as a part of the spatially varying range processing block 702, the HPC may vary one or more parameters of a range finding algorithm when processing low-level data received from different portions of the target scene, e.g., to capture certain spatial features of the target scene. In some embodiments, spatially varying range processing can improve resolution and correct range-dependent distortions. In some embodiments, spatially varying range processing can allow capturing certain spatial features such as background and / or clutter information with higher resolution because instead of applying the same transformation (e.g., a bin size for a dimension) to an entire frame (e.g., low-level radar data associated with the entire target scene), the HPC may judiciously adjust the transformation for different regions, e.g., by super sampling selected regions of the target scene to focus (e.g., regions of interest that may be specified based on reference data or identified by the HPC). In contrast, by using a fixed transformation over the entire frame, the existing radar data processing schemes may fail to capture or correct certain spatial features.

[0154] In some embodiments, as another part of the spatially varying range processing block 702, the HPC may use one or more learned window functions to process the ADC samples 604. For example, the HPC may use different learned windows (or window functions) for different portions of the signal that correspond to different spatial locations in the environment. As such, multiple window functions may be used to process the ADC samples 604. The window functions may be learned using a neural network trained using training radar data and annotated radar data, where the annotated radar data indicates a range or distance of objects within the training radar data. By spatially varying the window at different locations or ranges, the resolution of the radar signal may be better preserved. In some examples, learning a window function may comprise determining values of a parameter of the window function (e.g., a length, aslope, ashape, or the like) using a machine learning model trained based on a set input data and a corresponding set of output data (e.g., output data having desired features). In some embodiments, the output data may be generated, at least partially, using a known algorithm and specified parameters of the algorithm.

[0155] At the frequency varying Doppler processing block 704, the HPC may process the output of the spatial varying range processing block 702 to generate a range DopplerMOTN.153WO PATENTcube, also referred to herein as RDC 705. The RDC 705 may be implemented as a three-dimensional array representing a space-time processing of radar data. The RDC 705 may include two-dimensional or three-dimensional location information and / or velocity information for some or all points within the RDC 705.

[0156] Similar to the windows used in the spatial varying range processing block 702, the HPC may use one or more window functions (e.g., learned window functions) to process the low-level radar data to determine velocity of a point or an object in the target scene. Similar to the spatial varying range processing block 702, the number of windows and parameters for the windows may be learned offline (e.g., using an optimization algorithm, etc.). As such, during online processing, the HPC may select which windows to use for the signal (e.g., based on the frequency or other parameters of the signal).

[0157] As described herein, the output of the frequency varying Doppler processing block 704 may include one or more RDCs 705. In some embodiments, such as when a multiple-input multiple-output (MIMO) radar is used, the HPC may generate at least one RDC 705 for some or all antennas of the radar system (e.g., the antennas in the corresponding radar sensor). As such, the HPC may concurrently process multiple radar signals to generate at least one RDC 705 for some or all antennas of the radar system.

[0158] In certain cases, the HPC may use an FFT to convert time domain radar data (e.g., low-level radar data) to a frequency domain radar data. The HPC may then apply different window functions in frequency domain to the frequency domain radar data to determine one or both range and velocity of points and / or objects within the environment (e.g., target scene of the corresponding radar sensor). In certain cases, the HPC selects and combines the windows based on the frequency of a radar signal. In some embodiments, the HPC uses a fast Fourier transform (FFT) to convert the time domain radar data to the frequency domain radar data. The frequency domain may correspond to the spatial domain (range dimensions). In some embodiments, one or both spatially varying range processing and frequency varying Doppler processing may be performed in frequency domain.

[0159] The windows or window functions used for spatially varying range processing and frequency varying Doppler processing may be combined in a variety of ways. In some embodiments, the window functions are combined as a continuum of window functions.MOTN.153WO PATENTAs such, tens, hundreds, thousands, millions, or more window functions may be used to process the ADC samples 701 to determine the range and velocity of points and / or objects.

[0160] Usage of spatially varying range processing and frequency varying Doppler processing in the radar data processing pipeline 700 are enabled by implementing the corresponding blocks in the HPC having sufficient computation power to handle the computational load associated with these techniques. In some embodiments, it may not be feasible to implement techniques using the radar DSPs of the radar system 500 due computational limitations of the radar DSPs (e.g., lower processing power, slower execution, and smaller memory size).

[0161] In some embodiments, during an online processing period, the HPC judiciously varies one or more parameters used to determine range and / or velocity for objects and points at different locations in a target scene. In some such embodiments, the HPC can vary the one or more parameters based at least in part on a trained model (e.g., a learned optimization function). The trained mode may be trained offline using training radar data (e.g., training radar data collected and labeled during an offline training period).

[0162] In some embodiments, by placing the first semantic preserving filter 706 after the spatial varying range processing block 702 and frequency varying Doppler processing block 704, the HPC may produce higher fidelity radar images with more data points, thereby enabling a higher resolution radar image.

[0163] In some embodiments, at spatially varying range processing block 702, and frequency varying Doppler processing block 704, the HPC may use machine-learned range and velocity finding algorithms to determine a range or velocity. In some embodiments, at spatially varying range processing block 702, and frequency varying Doppler processing block 704, the HPC may use a weighted least square-based data-dependent, non-parametric algorithm, such as an iterative adaptive approach (IAA) and / or a transformer-based learned range and velocity finding algorithms to determine a range or velocity.

[0164] In some embodiments, at spatially varying range processing block 702, and / or frequency varying Doppler processing block 704, the HPC may vary, scan, or sweep at least one processing parameter to generate processed radar data comprising one orMOTN.153WO PATENTmore of a range or velocity of radar-scatterers in an environment. In some embodiments, the HPC may vary, scan, or sweep the processing parameter (e.g., the value of the processing parameter) to determine at least two different values of the processing parameter for each of the at least one of the range and velocity. In some embodiments, at spatially varying range processing block 702, and / or frequency varying Doppler processing block 704, the HPC may sweep a parameter by stepping the parameter through a range of values. In some examples, sweeping may be used to provide a bruteforce the solution (e.g., trying stepped values of the parameter until a solution is found). In various implementations, the range of parameter values in a sweeping process and the stepping behavior can be different depending on the specific parameter and a target of the processing. For example, the processing parameter can be length of a windowing function and an optimal value of the length can be determined by starting at first length and linearly stepping to a scone length while evaluating the resulting processed radar data for each value of the length of the windowing function.

[0165] In some embodiments, some embodiments, at spatially varying range processing block 702, and / or frequency varying Doppler processing block 704, the HPC may judiciously (e.g. using machine learning) determine a limited set of different values of a processing parameter and evaluate individual values in the limited set to determine the optimal value of the processing parameter in a shorter processing time and / or using less computational resources (e.g., compared to processing time and / or computational resources used in the sweeping process).

[0166] In some such embodiments, the HPC may use each one of the at least two different values of the processing parameter to determine the velocity of one of at least two radar-scatterers at two different positions in the environment (e.g., a scene in the environment monitored by the corresponding radar unit). For example, the HPC may use a first value of a processing parameter to determine a first range for a first object (e.g., a target object) in the environment and a second value of the processing parameter to determine a second range for a second object (e.g., a background object) in the environment. In some embodiments, the HPC may select the first and second values using a trained machine learning model such that the resulting processed radar data includes data points usable for generating a radar image comprising the images of theMOTN.153WO PATENTfirst and second objects. In some embodiments, the first and second values may be selected such that the images of the first and second objects have resolutions larger than a specified value. In some embodiments, the HPC may use a first set of values for given set of parameters to determine range (or velocity) for a first point in the environment and a second set of values for the set of parameters to determine range (or velocity) for a second point in the environment.

[0167] In some embodiments, the HPC may select and determine two different processing parameters (e.g., two parameters of different types) or two different sets of processing parameters (e.g., one set comprising at least one parameter not included in the other set) wherein for each of the at least one of the range and velocity. In some such embodiments, the HPC may use each one of the two different parameters or two different sets of processing parameters to determine the velocity of one of at least two radar-scatterers at two different positions in the environment or scene. For example, the HPC may use a first set of processing parameters to determine a first range for a first object (e.g., a target object) in the environment and a second set of processing parameters to determine a second range for a second object (e.g. a background object) in the environment. In some embodiments, the HPC may select the first and second sets of processing parameters using a trained machine learning model such that the resulting processed radar data includes data points usable for generating a radar image comprising images of the first and second objects images. In some embodiments, the first and second sets of parameters may be selected such the images of the first and second objects have resolutions or contrasts larger than a specified value.

[0168] In some implementations, the selected / determined processing parameter may comprise a type of a window function, a parameter of window function (e.g., a length, a shape, a slope, or the like), a bin size, or another parameter. In some embodiments, the selected / determined processing parameter may comprise a selected group of bins to which a window function is applied. For example, the HPC may apply a first window to bins 1 -10 and a second window to bins 11 -20.

[0169] In some embodiments, the first semantic preserving filter 706 may be used to remove noise from the RDC 705 and generate an RDC with fewer data points (also referred to herein as a sparse RDC 724). In some embodiments, the first semanticMOTN.153WO PATENTpreserving filter 706 may selectively sample processed radar data included in the RDC 705 to reduce computational complexity while maintaining accuracy and generate the sparse RDC 724. In some embodiments, the first semantic preserving filter 706 may accomplish this by performing a parametric or machine learning algorithms including, but not limited to: non-coherent integration, noise floor estimation, and noise floor thresholding.

[0170] In the illustrated example, the first semantic preserving filter 706 follows the spatial varying range processing block 702 and frequency varying Doppler processing block 704. Additionally, in some embodiments, the first semantic preserving filter 706 may remove sidelobes (e.g., corresponding to targets and / or clutter). In some embodiments, the first semantic preserving filter 706 may be configured to remove noise and sidelobes from radar data (e.g., associated with a reflected radio wave) to preserve semantic information included in clutter. For example, the first semantic preserving filter 706 may suppress side-lobes in the range and Doppler dimensions. In various implementations, the first semantic preserving filter 706 may remove a side-lobe using a windowing function or setting a threshold (e.g., amplitude threshold) as described below), or another method.

[0171] With continued reference FIG. 7A, at the multiple-input multiple-output (MIMO) processing block 708, the HPC may demultiplex processed radar data corresponding to different antennas of a corresponding radar sensor, from which the digitized low-level radar data 701 is received, to generate a virtual array response. In some embodiments, individual radar systems 520, 600 (e.g., the radar units 511a / b or 602-1 / 2, therein), may use multiple transmit and receive antennas to enable MIMO processing and thereby improve spatial resolution, target detection, and immunity of the radar system to interference. A MIMO based radar system may transmit mutually orthogonal signals, allowing for better time-bandwidth utilization, angle estimation, and target separation.

[0172] At the super-resolution angle finding block 710, the HPC may determine the angular location of points and objects in the environment (e.g., a target scene) with respect to the radar system (e.g., a corresponding radar sensor). In some embodiments, at the super-resolution angle finding block 710, the HPC may use super-resolution angle of arrival (AoA) estimation algorithms such as iterative adaptive approach (IAA) to determine the direction from which a reflected radio wave of a plurality of reflected radioMOTN.153WO PATENTwaves, propagating along closely spaced directions, has arrived at an antenna, with resolution beyond the classical Rayleigh limit. The angle finding algorithm used at the super-resolution angle finding block 710 may be learned using a transformer-based neural network architecture trained on the output of an AoA algorithm (e.g. IAA) on radar signals to learn how to estimate angle of arrival from incoming radar signals. In some embodiments, the use of super-resolution AoA algorithms and machine learning techniques in the super-resolution angle finding block 710 can improve target separation (e.g., improve resolution of angular target separation) compared to existing radar systems (such as radar system 500) that may be unable to apply similar methods due to the constraints of their DSP modules.

[0173] In some embodiments, at the super-resolution angle finding block 710, the HPC may use advanced angle finding methods including but not limited to the multiple signal classification (MUSIC) (e.g., based on eigenvalue decomposition to separate signals and noise), bandwidth extrapolation (BWE), extended coherent processing (ECP), and adaptive super-resolution reconstruction classes of algorithms.

[0174] The second semantic preserving filter 712 may be used to remove undesired sidelobes (e.g. sidelobes along the range, Doppler, azimuth, or elevation dimensions) typically associated with target objects and / or clutter. In some embodiments, the HPC may suppress the sidelobes associated with the radiation pattern of the corresponding antennas and, in some embodiments, improve the angular dynamic range (the difference of resolvable targets with max and min strength). In various implementations, the second semantic preserving filter 712 may remove a side-lobe using a windowing function or setting a threshold (e.g., amplitude threshold) as described below), or another method.

[0175] As described above, in some embodiments, spatially varying range processing block 702, frequency varying Doppler processing block 704, and super resolution angle finding block 710, may use machine learning neural networks for performing the corresponding processing tasks. In some embodiments, these machine learning methods may comprise a transformer-based model. In some embodiments, the iterative adaptive approach (IAA) algorithm may be used as baseline for the model. In such cases, instead of an explicitly programmed implementation of IAA, a model can be trained to perform IAA.MOTN.153WO PATENT

[0176] In some embodiments, the windowing functions (e.g., a parameter such a length of a windowing function) and other parameters used in various blocks of the radar data processing pipeline 700, may be swept, varied, or otherwise selected such that, as the low-level radar data is refined to allow more accurate range, velocity and angle processing, at least a portion of the information associated with background and clutter included in the low-level radar data (e.g., digitized low-level radar data 701) is preserved and processed to produce corresponding points in a point cloud or radar image output by the radar data processing pipeline 700. In some such embodiments, the information associated with background and clutter can be maintained using machine learning methods combined with dynamic and smart thresholding and side-lobe removal methods. Implementation of the radar data processing pipeline 700 in whole or in part on the HPC system (e.g., HPC system of a vehicle) enables the use of computationally demanding algorithms, substitution with machine learning algorithms, and fast and efficient processing of increased radar data volumes originating from the retained clutter and background information. For example, a windowing function (e.g., a type of the windowing function or a parameter of a windowing function) and / or other processing parameter used in various blocks of the radar data processing pipeline 700, may be selected a trained machine learning model trained using a set of input data and corresponding output data where the input data is collected during a testing / preparation period or generated based on previously collected radar data, and the output data is generated by processing the input data using a known algorithm and / or provided based on features of a desired radar image (e.g., a radar image comprising a background image and a target image). In some examples, a parameter of the known algorithm may be adjusted (e.g., during one or more iterations) such that at least a portion of radar data associated with background / clutter is maintained and / or contrast / resolution of a radar data associated with a target object.

[0177] In various implementations, range, elevation, azimuth and velocity, as the four separate dimensions of processed radar data, can each include main lobes and sidelobes (in spatial or frequency domains). In the spatial domain, the main lobe is the primary radiation pattern of a radar sensor or an antenna therein, where the majority of the transmitted power is concentrated and is directed towards a target. Side-lobes areMOTN.153WO PATENTsecondary radiation patterns that occur at angles away from the main lobe. Side-lobes are typically weaker than the main lobe but can cause interference and false detections.

[0178] Side-lobes in the spatial or frequency domains can interfere with the main lobes and degrade the accuracy and quality of a radar image. As such removal of the sidelobes may be a helpful part of radar data processing that can be handled using a combination of windowing, thresholding and other techniques. In some implementations, the sidelobes for range, elevation, azimuth and velocity can be suppressed separately. However, embodiments are not so limited, the sidelobes of two or more dimensions may be removed, at least partially, by a single side-lobe removal process.

[0179] In some embodiments, a windowing function may be used to suppress a sidelobe without affecting (or by minimally affecting) a corresponding main lobe (a main lobe associated with clutter or a target) in spatial or frequency domain. For example, a windowing function may be configured to attenuate / reduce amplitude of radar data in spatial (or frequency domain) corresponding to spatial (or frequency) points within a specified range near peak of a sidelobe and leave the spatial (or frequency) points within a specified range near peak of a main lobe (e.g. , a corresponding main lobe) substantially unaltered.

[0180] In some embodiments, thresholding may be used to suppress a sidelobe. For example, an amplitude threshold for transmitting radar data (e.g., in spatial or frequency domain) to a subsequent processing block may be configured such that radar data within a specified range near peak of a main lobe are transmitted and radar data within a specified range near peak of a sidelobe are eliminated.

[0181] In various implementations, signal processing techniques (e.g., filtering and windowing), and beamforming techniques can help reduce side-lobes. With reference to the radar data processing pipeline 700 in FIG. 7A, in some embodiments, the first semantic preserving filter 706 may be configured to suppress side lobes in range and velocity / Doppler dimensions, and the second semantic preserving filter 712 may be configured to suppress side lobes in azimuth and elevation dimensions. In various implementations, the first semantic filter may use windowing functions and / or thresholding to reduce a side-lobe (as described above). However, the embodiments are not so limited and other methods may be used to reduce or suppress a side-lobe in spatial or frequencyMOTN.153WO PATENT

[0182] In some embodiments, removal of side-lobes in the range and velocity dimensions may be performed, at least partially, by the spatially varying range processing block 702 and the frequency varying Doppler processing block 704. In some such cases, the side-lobes in the range and velocity dimensions can be frequency-domain side-lobes in a frequency domain representation of the low-level radar data and can be removed by applying window functions in frequency domain. In some embodiments, at the spatially varying range processing block 702 and the frequency varying Doppler processing block 704, a processor of the AV HPC 510 can use a machine learning model to generate learned window functions for side-lobe suppression.

[0183] FIG. 7B is a block diagram illustrating example processing blocks of the first semantic preserving filter 706 in the radar data processing pipeline 700. In the illustrated example, the first semantic preserving filter 706 may include a range Doppler (RD) accumulation block 716, a noise floor estimation block 720, and a noise thresholding block 718, and a dynamic range filtering block 722. The various processing blocks 716-722 of the first semantic preserving filter 706 may be implemented using one or more high performance computers implemented with one or more CPU (or GPUs) and non-transitory memories of a HPC system (e.g. AV HPC 510).

[0184] In some embodiments, the first semantic preserving filter 706 may receive the RDC 705, e.g., from the frequency varying Doppler processing block 704, provide a copy or a portion of the RDC 705 to the noise thresholding block 720 and another copy or a different portion of the RDC 705 to the RD accumulation block 716. At the RD accumulation block 716, the HPC may accumulate and non-coherently combine multiple RDCs 105 from multiple receivers / antennas of a radar sensor. In some embodiments, RD accumulation may comprise integrating multiple radar chirps / pulses over time to enhance signal strength, improve signal-to-noise ratio (SNR), and improve detection accuracy, especially for slow-moving or weak (low radar cross-section) targets.

[0185] In some embodiments, the HPC may accumulate the RDCs 705 in a buffer, volatile memory, and / or non-volatile memory, etc. The combined and / or accumulated RDCs may be provided to the noise floor estimation block 718 configured to estimate a noise floor and provide the estimated noise floor to the noise thresholding block 720. At the noise floor estimation block 718, the HPC estimates a noise floor of the partiallyMOTN.153WO PATENTprocessed radar data based on the accumulated RDCs received from the RD accumulation block 716. In some embodiments, the noise floor may be estimated based on a median of the signals from the accumulated RDCs 705. As the accumulated RDCs may represent a particular environment, the HPC may estimate the noise floor for the signal within this environment. In some examples, the noise floor may be estimated by determining the median value of amplitude of the signals from the accumulated RDCs 705 and radar data having an amplitude of less than a fraction (e.g., 1 / 10, 1 / 4, 1 / 2, or other values), of the median value may be labeled, or identified as noise.

[0186] At noise thresholding block 720, the HPC may determine one or more noise thresholds to apply to the RDCs 705. In some embodiments, the HPC may determine at least some of these noise thresholds based on the estimated noise floor received from the noise floor estimation block. The noise threshold(s) may then be applied to the radar signal or RDCs 705 to remove unwanted noise. For example, the noise threshold may comprise an amplitude substantially equal to the noise floor multiplied by a specified / determined value or equal to sum of the noise floor and a specified / determined value. In some cases, the specified / determined value may comprise a value provided by a user, received from a memory (e.g., selected from a set of stored values), or determined based at least in part on an average or medial value of the peak amplitudes associated with the reflected radar signals and the noise floor.

[0187] At the dynamic range filtering block 722, the HPC may dynamically filter the partially processed radar data received from the noise thresholding block 720 to suppress sidelobes corresponding to targets and / or clutter and generate the sparse RDC 724 output. As described above, in addition to reflections from objects in a corresponding target scene, relevant information may comprise reflections from a background or clutter. As such, the dynamic range filtering block 722 may be configured to preserve information associated with these background or clutter reflections while removing the portions of radar data that may degrade the accuracy and clarity of the objects in the resulting radar images. For example, the dynamic range filtering block 722 may identify and remove sidelobes corresponding to targets and / or clutter in order to achieve this. In some embodiments, the dynamic range filtering block 722 may vary the threshold or filter across multiple dimensions (e.g. range and / or Doppler) to more accurately differentiate main lobeMOTN.153WO PATENTreflections from sidelobes, while retaining radar signals corresponding to targets and clutter.

[0188] In certain cases, the HPC may dynamically select the strongest signal in each frame, determine the threshold for the frame, and remove or mask the signals below the threshold for that frame. In some embodiments, the HPC may combine multiple frames to determine the threshold and then apply the threshold to the signals in the corresponding frames.

[0189] In some embodiments, the first semantic preserving filter 706 can be configured to substantially preserve shape or a shape parameter of a radar-scatterer in the environment while filtering the RDC 705 output by the frequency varying Doppler processing block 704. As such, the processed data 614 (e.g., a radar image therein or generated using the processed data 614) may comprise, or may be used to generate, an image of the radar-scatterer having a shape or shape parameter close, similar, or substantially identical to that of the radar-scatterer (e.g., a shape or shape parameter in an optical image of the radar-scatterer). In various implementations, a shape parameter may comprise one or more ratios between two dimensions of the radar-scatterer (e.g., a width in a lateral direction, a length in a longitudinal direction, a height in vertical direction), a curvature, an outline, or other geometrical parameters that may be used to distinguish, identify, or characterize a radar-scatterer based on its radar image.

[0190] In some embodiments, the dynamic range filtering block 722 and the noise floor estimation block 718 can be configured so that the first semantic preserving filter 706 preserves a shape or shape parameter of a detected object target (e.g., radar-scatterer in the environment) in a target scene such that the shape or the shape parameter of a corresponding image portion of a radar image generated based on the output of the first semantic preserving filter 706, is close, similar, or substantially identical to that of the radar-scatterer or optical image of the radar-scatterer.

[0191] Advantageously, implementing the building blocks of the first semantic preserving filter 706 (e.g., dynamic range filtering block 722 and the noise floor estimation block 718) may allow adjusting the corresponding parameters such that the semantic information (e.g., information associated with shapes of the objects in the target scene) are preserved and thereby can be included in the resulting radar image. Given the largeMOTN.153WO PATENTvolume of radar data that should be maintained (e.g., by adjusting the noise thresholds and filters) to preserve the semantic information, it may not be feasible and practical to implement TO such semantic preserving filter in the radar DSPs of existing radar systems which include limited computational resources and memory.

[0192] FIG. 7C is a block diagram illustrating non-limiting example processing blocks within the second semantic preserving filter 712. In some embodiments, the second semantic preserving filter 712 may remove the side-lobes in the elevation and azimuth dimensions. In the illustrated example, the second semantic preserving filter 712 includes a sidelobe classification block 728 and a sidelobe removal block 730. The sidelobe classification block 728 may be used to identify / classify sidelobes within (e.g., within the sparse range elevation azimuth Doppler (READ) cubes 726, and the sidelobe removal block 730 may be used to remove the sidelobes from the signal. In some embodiments, at the sidelobe classification block 728, the HPC may use angle-based classification, power-level analysis, and / or other methods to classify sidelobes.

[0193] At the sidelobe classification block 728, the HPC may use a super-resolution angle finding algorithm such as IAA to suppress the sidelobe level within the input array vector. The HPC may then use a learned algorithm to assign each peak a confidence score which corresponds to the probability that the peak is a main lobe that corresponds to a target or clutter, as opposed to a sidelobe (of a target or clutter).

[0194] Although described in terms of a confidence level that a peak is a main lobe, other configurations may be used. For example, the HPC may use a confidence level that indicates the probability that a particular peak is a sidelobe.

[0195] At the sidelobe removal block 730, the HPC may remove a peak or portion of the signal based on the confidence level. For example, if a confidence value of a peak is assigned a confidence score that does not satisfy (e.g., is at or below a predetermined threshold, the HPC may remove the peak. The threshold may be determined by maximizing the f1 -score on the evaluation set. In some examples, the HPC may remove the peak by replacing the radar data associated with the peak (e.g., spatial or frequency domain radar data within a range around a spatial parameter or the frequency at which the peak occurs) by a specified value (e.g., a value substantially equal to a measured or determined noise floor).MOTN.153WO PATENT

[0196] As described herein, a confidence level that does not satisfy the threshold may indicate that the peak is likely a sidelobe. Accordingly, by removing peaks / portions of the signal that do not satisfy the threshold, the HPC may remove peaks or portions of the signal that correspond to sidelobes, while retaining peaks that correspond to main lobes of a target or clutter.

[0197] The output of the sidelobe removal block 730 and / or second semantic preserving filter 712 may include a higher resolution radar image that includes main lobes corresponding to targets and clutter in an environment. By retaining data (e.g., main lobes) corresponding to clutter, the perception radar imaging pipeline 700 may improve the ability of a perception system to perceive an environment and the objects within the environment, which can lead to improved navigation for an autonomous vehicle

[0198] In some embodiments, the first and second semantic preserving filters 706, 712, may be configured to enhance the quality of the radar image 714 while maintaining at least a portion of radar data associated with clutter and maintaining semantic integrity by preserving important structures and details, while filtering out noise, unwanted artifacts, and / or side-lobes. Advantageously, implementing the first and second semantic preserving filters 706, 712, in the HPC system enables usage of computationally-demanding signal processing and machine learning techniques that were not feasible in existing radar systems to maintain and process clutter while preserving semantic integrity. This may include dynamically estimating and applying one or more thresholds for noise or sidelobe energy across one or more dimensions (e.g. range, Doppler, azimuth, or elevation).

[0199] In various implementations, machine learning methods may be used to implement one or more processing or computing tasks in the radar data processing pipeline 700. A machine learning model may accomplish a processing to computing task by being trained on a large body of input and output data so that it learns patterns that may map input data to output or processed data. After learning these patterns, the machine learning model may be sued to process new input data by applying the learned patterns to produce corresponding processed data.

[0200] In some embodiments, the radar data processing pipeline 700 may be configured to reduce the size and / or volume of the processed data with minimalMOTN.153WO PATENTinformation loss. For example, one or more machine learning algorithms used in the radar data processing pipeline 700 (e.g., within one of the blocks) may be configured to preserve more information in the processed data compared to a radar DSP module (e.g., radar DSP modules 504a or 504b) while reducing the size and / or volume of the processed data.

[0201] In some embodiments, the radar data processing pipeline 700 may use a machine learning modelto perform at least a portion of a processing task comprising sidelobe reduction and / or clutter preservation. In some embodiments, the processing task comprising side-lobe reduction and / or clutter preservation can be broken down into spatially varying range processing (e.g., performed spatially varying range processing block 702) and frequency varying Doppler processing (e.g., frequency varying Doppler processing block 704), or other processing tasks in the radar data processing pipeline 700.In some embodiments, a processing task can be tackled by a machine learning model trained using a variety of radar signal processing algorithms (e.g., known and existing algorithms), which may rely on explicitly programmed parameters or heuristics. For example, a machine learning model may be trained by 1) collecting a large body of input data, 2) using radar signal processing algorithm, to process the input data and generate output (processed) data, and 3) training the machine learning (ML) algorithm to find and record the patterns that map the input to the output. The trained ML algorithm may be used to perform the corresponding task in the radar data processing pipeline 700. In some examples, the trained ML model can emulate and / or replicate (at least approximately) the radar signal processing algorithm functionality on new input data (e.g., real-time data (e.g., low-level data) received from the a radar sensor or an intermediate data received from a previous processing block. In some embodiments, the trained ML algorithm may achieve some faster runtime with equivalent accuracy, equivalent runtime with better accuracy, or faster runtime with better accuracy, compared to the radar signal processing algorithm itself.

[0202] As described herein, a reflected radar signal and thereby the digitized low-level radar data 701 may include data corresponding to clutter (environment) and targets (objects within the environment). The reflected radio waves thereby the digitized low-levelMOTN.153WO PATENTradar data 701 and corresponding frequency domain radar data (e.g., generated by transforming the digitized low-level radar data 701) can include main lobes and sidelobes associated with clutter and targets. In certain cases, a sidelobe may correspond to an additional peak in any one of the dimensions (range, Doppler / velocity, azimuth or elevation) of a radar cube other than main peaks corresponding to the reflection of a main lobe of a radiation pattern emitted by the corresponding radiator sensor.

[0203] A main lobe may correspond to reflection of a main lobe of a radiation pattern emitted by the radar sensor by a target or background. Sidelobes in spatial domain (e.g., side-lobes of low-level radar data) may be the result of interaction between a side-lobe of an emission pattern with a target or background. Put another way, sidelobes may correspond to an undesired ripple of the target beside the main lobe after signal processing. The height and location of sidelobes may be a complex function of multiple factors, such as but not limited to, target response, scene configuration, and device characteristics. FIG. 8A schematically illustrates multi-lobe emission pattern 804 emitted by a radar antenna 802, the reflected radio waves 808, 810 generated as a result of interaction of the main lobe 806 with a target 814 and background 816, respectively, and reflected radio waves 809, 812 generated as a result of interaction of the side-lobes 805a, 805b with the target 814 and background 816, respectively.

[0204] In some embodiments, when low-level radar data is transformed from the temporal or spatial domain to the frequency domain, e.g., for spatially varying range processing and / or frequency varying Doppler processing, the resulting radar data may include frequency domain main lobes and sidelobes. In some embodiments, the frequency domain sidelobes may correspond to specific temporal characteristics of a corresponding emitted radio wave, certain parameters of the transformation (e.g., the FFT), windowing functions, as well as side-lobes in the antenna pattern. FIG. 8B schematically illustrates an example frequency spectrum of the reflected radar signals or frequency domain representation of low-level radar data, e.g., corresponding to reflected radio waves shown in FIG. 8A. In this example, the spectrum includes a target main lobe 818, a plurality of target side-lobes 819, a clutter main lobe 820, and a plurality of clutter side-lobes 821.MOTN.153WO PATENT

[0205] In some embodiments, to filter sidelobes of the target or clutter, the HPC may dynamically adjust its threshold based on the incoming signal. For example, within a frame of data (e.g., one or more RPCs 705 associated with a particular time or time range), the HPC may determine a threshold based on the main lobe of the target or clutter. In some embodiments, HPC may determine the threshold based on the peak or maximum of the main lobe of the target or clutter. For example, the HPC may determine the highest value of the signal in each range dimension, fix the range beam and look at the Doppler dimension, and set a threshold based on the determined value(s). In some embodiments, the threshold may be set at some dB below a portion of the main lobe, such as the maximum or peak of the main lobe. For example, the HPC may set the threshold at 20-30 dB below the maximum of the main lobe of a target in the Doppler dimension. Waves below the threshold may be masked or removed

[0206] In some embodiments, a threshold may be set to remove sidelobes, while retaining some or all main lobes regardless of whether the main lobes correspond to a target or to clutter.

[0207] In some embodiments, the HPC may use different thresholds for different domains. For example, the HPC may use one threshold in the Doppler (velocity) domain and a different threshold in the spatial domain

[0208] By dynamically or adaptively thresholding or applying a filter to the radar signal to remove sidelobes while retaining main lobes (of targets and / or clutter) based on factors determined or estimated from the radar signals, the dynamic range filtering block 722 may preserve the full dynamic range of the system and improve radar image resolution of a particular environment.

[0209] In various embodiments described above, a machine learning model used to perform a processing task (e.g. , a machine learning model used in a block of a processing pipeline) can be a trained machine learning model trained using input and output data generated during a testing / preparation period. In some embodiments, the input and output data may be provided by collecting a large body of input data collected and / or generated by a radar system (e.g., by the radar unit) during the testing / preparation period, or generated based on previously collected radar data and using a radar signal processing algorithm, to process the input data and generate output (processed) data. InMOTN.153WO PATENTsome embodiments, the radar signal processing algorithms can be a known / existing algorithm), which may rely on explicitly programmed parameters or heuristics.Radar Perception Model Consuming Low-Level Radar Image

[0210] The architecture previously described constitutes a fully Software Defined Radar (SDR) system combined with an end-to-end machine learning pipeline, enabling ongoing innovation throughout the entire lifecycle of the autonomous vehicle (AV) platform. To fuel this vision, the system uses data sets to learn.

[0211] A multi-modality data set may be used that integrates low-level radar output with camera and LiDAR data. The dataset may include 1 petabyte of sensor data, for training and refining machine learning algorithms. The dataset may be further enriched with detailed annotations generated through a scalable pipeline that employs both automated labeling and manual methods. This vast and comprehensive dataset may be used to iteratively enhance the sensing and perception pipeline, thereby accelerating the pace at which a vehicle learns.

[0212] The radar systems 520 and 600, and the radar data processing pipeline 700 described above may be used to generate higher resolution radar images of an environment that in addition to targets include clutter.

[0213] FIG. 9A is an image showing a row of vehicles parked along a curb identified by a perception system as depicted by bounding boxes.

[0214] FIG. 9B is a top view 2D radar image of the scene (light gray regions) captured in the image shown in FIG. 9A generated by an existing imaging radar system overlapped with a top view optical image of the scene. In FIG. 9B the detected vehicles are depicted by dashed boxes. In this radar image, some portions of the street curb 902 behind some of the vehicles is not imaged or detected and the vehicles have few or no data points. For example, an individual vehicle 904 is identified on the radar image based on small regions that provide limited information about the shape and / or dimension of the vehicle.

[0215] FIG. 9C is a top view 2D radar image of the scene captured in the radar image shown in FIG. 9B generated by the disclosed imaging radar systems (e.g., using the enhanced radar system 600 and the radar data processing pipeline 700), providing a detailed view of the roadway and the vehicles. In comparison with the 2D radar imageMOTN.153WO PATENTshown in FIG. 9B, at least a portion of the street curb 902 is clearly captured with high contrast, as depicted with the thick line, and most vehicles are imaged in a more complete, detailed view in the 2D radar image shown in FIG. 9C (captured by an enhanced radar system). For example, vehicle 904 is depicted by a large region with high contrast that better represents the shape and dimensions of the vehicle.Example Embodiments

[0216] Example embodiments described herein have several features, no single one of which is indispensable or solely responsible for their desirable attributes. A variety of example systems and methods are provided below.

[0217] Example 1. A radar system comprising: a radar unit configured to generate radar data in response to receiving radio waves reflected by radar-scatterers in an environment; a computing system in data communication with the radar unit, the computing system comprising a processor configured to execute machine-readable instructions to process the radar data and generate processed radar data comprising at least one of a range and a velocity of the radar-scatterers by varying a processing parameter to determine at least two different values of the processing parameter for each of the at least one of the range and the velocity.

[0218] Example 2. The radar system of Example 1, wherein computing system is configured to use each one of the at least two different values of processing parameters to determine the range of one of at least two radar-scatterers at two different positions in the environment.

[0219] Example 3. The radar system of any one of Examples 1-2, wherein computing system is configured to use each one of the at least two different values of processing parameters to determine the velocity of one of at least two radar-scatterers at two different positions in the environment.

[0220] Example 4. The radar system of any one of Examples 1-3, wherein the radar data is generated by at least two channels of the radar unit in response to emission at least two different radiation patterns by the radar unit.

[0221] Example 5. The radar system of any one of Examples 1-4, wherein the data communication of the radar unit and the computing unit is implemented at least in partMOTN.153WO PATENTby a serial data link, and the radar unit comprises a serializer that serializes the radar data for transition through the serial data link.

[0222] Example 6. The radar system of Example 5, further comprising a deserializer configured to receive the serialized data and provides radar data to the computing system.

[0223] Example 7. The radar system of Example 5, wherein the data link is configured to transmit data at a rate equal or greater than 3 Gigabit per second.

[0224] Example 8. The radar system of any one of Examples 1-7, wherein the radar unit is separated from the computing system

[0225] Example 9. The radar system of Example 8, wherein the computing system is configured to receive the radar data from the radar unit via a wireless link.

[0226] Example 10. The radar system of Example 8, wherein the computing system is configured to receive the radar data from the radar unit via a coaxial cable.

[0227] Example 11. The radar system of any one of Examples 1 -10, wherein the radar unit does not use a digital signal processing circuit to generate radar data.

[0228] Example 12. The radar system of any one of Examples 1-11, wherein the processor uses a trained machine learning model to vary the processing parameter.

[0229] Example 13. The radar system of Example 12, wherein the machine learning model comprises a transformer-based algorithm.

[0230] Example 14. The radar system of Example 12, wherein the machine learning model is trained based on outputs generated by a non-parametric algorithm.

[0231] Example 15. The radar system of any one of Examples 1-14, wherein the radar data comprises reflected radar signals generated by a transceiver of the radar unit.

[0232] Example 16. The radar system of Example 15, wherein radar data comprises digitized data output by an analog-to-digital converter of the radar unit in response to receiving reflected radar signals from the transceiver.

[0233] Example 17. The radar system of Example 15, further comprising a second radar unit in data communication with the computing system, wherein the second radar unit transmits second radar data comprising second reflected radar signals generated by a second transceiver of the second radar unit.MOTN.153WO PATENT

[0234] Example 18. The radar system of any one of Examples 1-17, wherein the processor further executes machine readable instructions to provide a first sematic preserving filter configured to filter the processed radar data to suppress one or both noise and side-lobe data.

[0235] Example 19. The radar system of Example 18, wherein the first side-lobe data comprises side-lobes in one or both range and velocity dimensions.

[0236] Example 20. The radar system of Example 18, wherein the first sematic preserving filter comprises a noise floor estimation module configured to estimate a noise floor of the processed radar data and a dynamic range filtering module configured to adaptively adjust dynamic range of the processed radar data.

[0237] Example 21. The radar system of Example 20, wherein the noise floor estimation module and the dynamic range filtering module are configured to preserve a shape of a radar-scatterer.

[0238] Example 22. The radar system of any one of Examples 1 -21 , wherein the radar- scatterers include a least one target object and at least one background, and the radar data include target data generated by the target object and clutter data generated by the background.

[0239] Example 23. The radar system of Example 22, wherein the computing system is configured to retain at least a portion of clutter data when processing the target data.

[0240] Example 24. The radar system of Example 22, wherein the processor is configured to generate the processed radar data using both target data and clutter data.

[0241] Example 25. The radar system of Example 23, wherein the processor is configured to retain at least a portion of clutter data when processing the target data at least in part by varying the at least one processing parameter.

[0242] Example 26. The radar system of Example 23, wherein the processor is configured to retain at least a portion of clutter data when processing the target data based at least in part by using a machine learning model.

[0243] Example 27. The radar system of Example 23, wherein the processor is configured to use the processed radar data to generate a radar image comprising the target object and the background.MOTN.153WO PATENT

[0244] Example 28. The radar system of Example 27, wherein the processor further executes machine readable instructions to provide a second sematic preserving filter to remove side-lobes prior to generation of the radar image.

[0245] Example 29. The radar system of Example 28, wherein the side-lobes are in one or both elevation and azimuth dimensions.

[0246] Example 30. The radar system of any one of Examples 1-29, wherein the processor executes the machine-readable instructions to provide a spatially varying range processing module configured to process the radar data received from the radar unit to determine a range, and wherein the processing parameter comprises a range processing parameter.

[0247] Example 31. The radar system of Example 30, wherein the processor executes machine readable instructions to provide a frequency varying Doppler processing module configured to process an output of the spatially varying range processing module to determine a velocity, and wherein varying the processing parameter comprises a Doppler processing parameter.

[0248] Example 32. The radar system of any one of Examples 30 and 31 , wherein the processing parameter comprises a window function or a bin size.

[0249] Example 33. The radar system of any one of Examples 30 and 31 , wherein the processing parameter comprises a selected group of bins to which a window function is applied.

[0250] Example 34. The radar system of any one of Examples 1-33, wherein the processor executes machine readable instructions to provide a super-resolution angle finding module configured to determine an angle of a radar scatterer with respect to the radar unit, by processing output of a multiple input multiple output (MIMO) processing module provided by the processor.

[0251] Example 35. The radar system of Example 34, wherein the super-resolution angle finding module comprises a transformer-based machine learning model trained using the outputs of an iterative adaptive approach algorithm to determine the angle.

[0252] Example 36. A radar system comprising: a radar unit configured to receive reflected radio waves from an environment and generate radar data in response to receiving the radio waves reflected by radar-scatterers in the environment; aMOTN.153WO PATENTcomputing system in data communication with the radar unit, the computing system comprising a processor configured to execute machine readable instructions to process the radar data via a data link and generate processed radar data comprising one or more of a range, velocity, elevation, and azimuth of the radar-scatterers, wherein the radar-scatterers include at least one target object and at least one background, wherein the radar data includes target data generated by the target object and clutter data generated by the background, and wherein the processor is configured to use a machine learning model to generate the processed radar data using both target data and clutter data

[0253] Example 37. The radar system of Example 36, wherein the processor is configured to retain at least a portion of clutter data when processing the target data at least in part by using a machine learning model.

[0254] Example 38. The radar system of any one of Examples 36-37, wherein the radar data is generated by at least two channels of the radar unit in response to emission at least two different radiation patterns by the radar unit.

[0255] Example 39. The radar system of any one of Examples 36-38, wherein the at least one background comprises a static object.

[0256] Example 40. The radar system of any one of Examples 36-39, wherein the radar unit does not use a digital signal processing circuit to generate the radar data.

[0257] Example 41. The radar system of Example 38, wherein the at least one background one or more of a building, a road surface, a curb, a post, or a tree.

[0258] Example 42. The radar system of any one of Examples 36-41 , wherein the data communication of the radar unit and the computing unit is implemented at least in part by a serial data link, and the radar unit comprises a serializer that serializes the radar data for transition through the serial data link.

[0259] Example 43. The radar system of Example 42, wherein the data link is configured to transmit data at a rate equal or greater than 3 Gigabit per second.

[0260] Example 44. The radar system of any one of Examples 36-43, wherein the radar unit is separated from the computing system.

[0261] Example 45. The radar system of Example 44, wherein the computing system is configured to receive the radar data from the radar unit via a wireless link.MOTN.153WO PATENT

[0262] Example 46. The radar system of Example 44, wherein the computing system is configured to receive the radar data from the radar unit via a coaxial cable.

[0263] Example 47. The radar system of any one of Examples 36, wherein the processor is configured to execute machine- readable instructions to process the radar data and generate processed radar data by varying a processing parameter to determine at least two different values processing parameters for each of the at least one of the range and the velocity.

[0264] Example 48. The radar system of Example 47, wherein the processor uses a machine learning model to vary the processing parameter.

[0265] Example 49. The radar system of Example 48, wherein the machine learning model is trained based on outputs generated by a non-parametric algorithm.

[0266] Example 50. The radar system of Example 47, wherein computing system is configured to use each one of the at least two different values processing parameters to determine range of one of at least two radar-scatterers at two different positions in the environment.

[0267] Example 51. The radar system of Example 47, wherein computing system is configured to use each one of the at least two different values processing parameters to determine velocity of one of at least two radar-scatterers at two different positions in the environment.

[0268] Example 52. The radar system of Example 47 wherein the processor is configured to retain at least a portion of clutter data when processing the target data at least in part by varying the processing parameter.

[0269] Example 53. The radar system of any one of Examples 36-52, wherein the radar data comprises reflected radar signals generated by a transceiver of the radar unit.

[0270] Example 54. The radar system of Example 53, wherein radar data comprises digitized data output by an analog-to-digital converter of the radar unit in response to receiving reflected radar signals from the transceiver.

[0271] Example 55. The radar system of Example 53, further comprising a second radar unit in data communication with the computing system, wherein the secondMOTN.153WO PATENTradar unit transmits second radar data comprising second reflected radar signals generated by a second transceiver of the second radar unit.

[0272] Example 56. The radar system of any one of Examples 36-55, wherein the processor further executes machine readable instructions to provide a first sematic preserving filter configured to filter the processed radar data to suppress at least one of noise and side-lobe data.

[0273] Example 57. The radar system of Example 56, wherein the first side-lobe data comprises side-lobes in one or both range and velocity dimensions.

[0274] Example 58. The radar system of Example 56, wherein the first sematic preserving filter comprises a noise floor estimation module configured to estimate a noise floor of the processed radar data and a dynamic range filtering module configured to adaptively adjust dynamic range of the processed radar data.

[0275] Example 59. The radar system of Example 58, wherein the noise floor estimation module and the dynamic range filtering module are configured to preserve a shape of a radar-scatterer.

[0276] Example 60. The radar system of any one of Examples 36-59, wherein the radar-scatterers include a least one target object and at least one background, and the radar data include target data generated by the target object and clutter data generated by the background.

[0277] Example 61. The radar system of Example 60, wherein the computing system is configured to retain at least a portion of clutter data when processing the target data.

[0278] Example 62. The radar system of Example 60, wherein the processor is configured to generate the processed radar data using both target data and clutter data.

[0279] Example 63. The radar system of any one of Examples 36-62, wherein the processor is configured to use the processed radar data to generate a radar image comprising the target object and the background.

[0280] Example 64. The radar system of Example 63, wherein the processor further executes machine readable instructions to provide a second sematic preserving filter to remove side-lobes prior to generation of the radar image.MOTN.153WO PATENT

[0281] Example 65. The radar system of Example 64, wherein the side-lobes are in one or both elevation and azimuth dimensions.

[0282] Example 66. The radar system of Example 47, wherein the processor executes the machine-readable instructions to provide a spatially varying range processing module configured to process the radar data received from the radar unit to determine a range, and wherein the processing parameter comprises a range processing parameter

[0283] Example 67. The radar system of Example 66, wherein the processor executes machine readable- instructions to provide a frequency varying Doppler processing module configured to process an output of the spatially varying range processing module to determine a velocity, and wherein varying the processing parameter comprises a Doppler processing parameter.

[0284] Example 68. The radar system of any one of Examples 66 and 67, wherein the processing parameter comprise a window function or a bin size.

[0285] Example 69. The radar system of any one of Examples 66 and 67, wherein the processing parameter comprises a selected group of bins to which a window function is applied.

[0286] Example 70. The radar system of any one of Examples 36-69, wherein the processor executes machine readable instructions to provide a super-resolution angle finding module configured to determine an angle of a radar scatterer with respect to the radar unit, by processing output of a multiple input multiple output (MIMO) processing module provided by the processor.

[0287] Example 71. The radar system of Example 70, wherein the super-resolution angle finding module comprises a transformer-based machine learning model trained using the outputs of an iterative adaptive approach algorithm to determine the angle.

[0288] Example 72. A method comprising: generating, by a radar unit, radar data in response to receiving reflected radio waves by radar-scatterers in an environment; receiving, by a processor of a computing system, the radar data from the radar unit via a data link; processing the radar data, by the processor, to generate processed radar data comprising one or more of a range, velocity, elevation, and azimuth of the radar-scatterers, wherein processing the radar data comprises varying a processingMOTN.153WO PATENTparameter to determine at least two different values of the processing parameters for each of the at least one of the range and the velocity.

[0289] Example 73. The method of Example 72, further comprising using each one of the at least two different values of processing parameters to determine the range of one of at least two radar-scatterers at two different positions in the environment

[0290] Example 74. The method of any one of Examples 72-73, further comprising using each one of the at least two different values of processing parameters to determine the velocity of one of at least two radar-scatterers at two different positions in the environment.

[0291] Example 75. The method of any one of Examples 72-74, wherein generating radar data comprises generating radar data by at least two channels of the radar unit by emitting at least two different radiation patterns by the radar unit.

[0292] Example 76. The method of any one of Examples 72-75, wherein the data link comprises a serial data link, and the radar unit comprises a serializer that serializes the radar data for transition through the serial data link.

[0293] Example 77. The method of Example 76, further comprising using a deserializer to receive the serialized data and to provide radar data to the computing system.

[0294] Example 78. The method of Example 76, wherein the data link is configured to transmit data at a rate equal or greater than 3 Gigabit per second

[0295] Example 79. The method of Example 72, wherein the radar unit is separated from the computing system.

[0296] Example 80. The method of Example 79, wherein the data link comprises wireless link.

[0297] Example 81. The method of Example 79, wherein the data link comprises a coaxial cable.

[0298] Example 82. The method of any one of Examples 72-81, wherein the generating radar data does not include digital signal processing.

[0299] Example 83. The method of any one of Examples 72-82, wherein varying the processing parameter comprises using a trained machine learning model.MOTN.153WO PATENT

[0300] Example 84. The method of Example 83, wherein the machine learning model comprises a transformer-based algorithm.

[0301] Example 85. The method of Example 83, wherein the machine learning model is trained based on outputs generated by a non-parametric algorithm.

[0302] Example 86. The method of any one of Examples 72-85, wherein the radar data comprises reflected radar signals generated by a transceiver of the radar unit.

[0303] Example 87. The method of Example 86, wherein radar data comprises digitized data output by an analog-to-digital converter of the radar unit in response to receiving reflected radar signals from the transceiver.

[0304] Example 88. The method of Example 86, further comprising generating, by a second radar unit, second radar data in response to receiving second reflected radio waves by the radar-scatterers in the environment.

[0305] Example 89. The method of any one of Examples 72-88, further comprising, using the computing system to provide a first sematic preserving filter configured to filter the processed radar data to suppress at least one of noise and side-lobe data.

[0306] Example 90. The method of Example 89, wherein the first side-lobe data comprises side-lobes in one or both range and velocity dimensions.

[0307] Example 91. The method of Example 89, wherein the first sematic preserving filter comprises a noise floor estimation module configured to estimate a noise floor of the processed radar data and a dynamic range filtering module configured to adaptively adjust dynamic range of the processed radar data.

[0308] Example 92. The method of Example 91, wherein the noise floor estimation module and the dynamic range filtering module are configured to preserve a shape of a radar-scatterer.

[0309] Example 93. The method of any one of Examples 72-91, wherein the radar- scatterers include a least one target object and at least one background, and the radar data include target data generated by the target object and clutter data generated by the background

[0310] Example 94. The method of Example 93, further comprising retaining at least a portion of clutter data when processing the target data.MOTN.153WO PATENT

[0311] Example 95. The method of Example 93, wherein generating the processed radar data comprises processing both target data and clutter data.

[0312] Example 96. The method of Example 94, wherein retaining at least a portion of clutter data comprises varying the at least one processing parameter.

[0313] Example 97. The method of Example 94, wherein retaining at least a portion of clutter data comprises using a machine learning model.

[0314] Example 98. The method of Example 94, further comprising generating a radar image comprising the target object and the background.

[0315] Example 99. The method of Example 98, further comprising using the computing system to provide a second sematic preserving filter configured to remove side-lobes prior to generation of the radar image.

[0316] Example 100. The method of Example 99, wherein the side-lobes are in one or both elevation and azimuth dimensions.

[0317] Example 101. The method of any one of Examples 72-100, further comprising using the computing system to provide a spatially varying range processing module configured to process the radar data received from the radar unit to determine a range, and wherein the processing parameter comprises a range processing parameter.

[0318] Example 102. The method of Example 101, further comprising using the computing system to provide a frequency varying Doppler processing module configured to process an output of the spatially varying range processing module to determine a velocity, and wherein varying the processing parameter comprises a Doppler processing parameter.

[0319] Example 103. The method of any one of Examples 101 and 102, wherein the processing parameter comprises a window function or a bin size.

[0320] Example 104. The method of any one of Examples 101 and 102, wherein the processing parameter comprises a selected group of bins to which a window function is applied.

[0321] Example 105. The method of any one of Examples 72-104, further comprising using the computing system to provide a super-resolution angle finding module configured to determine an angle of a radar scatterer with respect to the radar unit, byMOTN.153WO PATENTprocessing output of a multiple input multiple output (MIMO) processing module provided by the processor.

[0322] Example 106. The method of Example 105, wherein the super-resolution angle finding module comprises a transformer-based machine learning model trained using the outputs of an iterative adaptive approach algorithm to determine the angle.

[0323] Example 107. At least one non-transitory storage media storing machineexecutable instructions that, when executed by at least one processor of a computing system, causing the at least one processor to: receive radar data from a radar unit via a data link; process the radar data to generate processed radar data comprising one or more of a range, velocity, elevation, and azimuth of radar-scatterers in an environment, by varying a processing parameter to determine at least two different values of the processing parameter for each of the at least one of the range and the velocity.

[0324] Example 108. The least one non-transitory storage media of Example 107, wherein the at least one processor is configured to use each one of the at least two different values of processing parameters to determine the range of one of at least two radar-scatterers at two different positions in the environment.

[0325] Example 109. The least one non-transitory storage media of any one of Examples 107-108, wherein the at least one processor is configured to use each one of the at least two different values of processing parameters to determine the velocity of one of at least two radar-scatterers at two different positions in the environment

[0326] Example 110. The least one non-transitory storage media of any one of Examples 107-109, wherein the radar data is generated by at least two channels of the radar unit in response to emission at least two different radiation patterns by the radar unit.

[0327] Example 111. The least one non-transitory storage media of any one of Examples 107-110, wherein the data link comprises a serial data link, and the radar unit comprises a serializer that serializes the radar data for transition through the serial data link.MOTN.153WO PATENT

[0328] Example 112. The least one non-transitory storage media of Example 111, wherein the data link is configured to transmit data at a rate equal or greater than 3 Gigabit per second.

[0329] Example 113. The least one non-transitory storage media of any one of Examples 107-112, wherein the radar unit is separated from the computing system.

[0330] Example 114. The least one non-transitory storage media of Example 113, wherein the computing system is configured to receive the radar data from the radar unit via a wireless link.

[0331] Example 115. The least one non-transitory storage media of Example 113, wherein the computing system is configured to receive the radar data from the radar unit via a coaxial cable.

[0332] Example 116. The least one non-transitory storage media of any one of Examples 107-115, wherein the radar unit does not use a digital signal processing circuit to generate radar data.

[0333] Example 117. The least one non-transitory storage media of any one of Examples 107-116, wherein the at least one processor uses a trained machine learning model to vary the processing parameter.

[0334] Example 118. The least one non-transitory storage media of Example 117, wherein the machine learning model comprises a transformer-based algorithm.

[0335] Example 119. The least one non-transitory storage media of Example 117, wherein the machine learning model is trained based on outputs generated by a nonparametric algorithm.

[0336] Example 120. The least one non-transitory storage media of any one of Examples 107-119, wherein the radar data comprises reflected radar signals generated by a transceiver of the radar unit.

[0337] Example 121. The least one non-transitory storage media of Example 120, wherein radar data comprises digitized data output by an analog-to-digital converter of the radar unit in response to receiving reflected radar signals from the transceiver.

[0338] Example 122. The least one non-transitory storage media of Example 120, further comprising a second radar unit in data communication with the computing system, wherein the at least one processor is configured to receive second radar dataMOTN.153WO PATENTfrom the second radar unit, second radar data comprising second reflected radar signals generated by a second transceiver of the second radar unit.

[0339] Example 123. The least one non-transitory storage media of any one of Examples 107-122, wherein the at least one processor is further configured to provide a first sematic preserving filter configured to filter the processed radar data to suppress one or both noise and side-lobe data.

[0340] Example 124. The least one non-transitory storage media of Example 123, wherein the first side-lobe data comprises side-lobes in one or both range and velocity dimensions.

[0341] Example 125. The least one non-transitory storage media of Example 123, wherein the first sematic preserving filter comprises a noise floor estimation module configured to estimate a noise floor of the processed radar data and a dynamic range filtering module configured to adaptively adjust dynamic range of the processed radar data.

[0342] Example 126. The least one non-transitory storage media of Example 125, wherein the noise floor estimation module and the dynamic range filtering module are configured to preserve a shape of a radar-scatterer.

[0343] Example 127. The least one non-transitory storage media of any one of Examples 107-126, wherein the radar-scatterers include a least one target object and at least one background, and the radar data include target data generated by the target object and clutter data generated by the background.

[0344] Example 128. The least one non-transitory storage media of Example 127, wherein the at least one processor is configured to retain at least a portion of clutter data when processing the target data.

[0345] Example 129. The least one non-transitory storage media of Example 127, wherein the at least one processor is configured to generate the processed radar data using both target data and clutter data.

[0346] Example 130. The least one non-transitory storage media of Example 128, wherein the at least one processor is configured to retain at least a portion of clutter data when processing the target data at least in part by varying the at least one processing parameter.MOTN.153WO PATENT

[0347] Example 131. The least one non-transitory storage media of Example 128, wherein the at least one processor is configured to retain at least a portion of clutter data when processing the target data based at least in part by using a machine learning model.

[0348] Example 132. The least one non-transitory storage media of Example 128, wherein the at least one processor is configured to use the processed radar data to generate a radar image comprising the target object and the background.

[0349] Example 133. The least one non-transitory storage media of Example 132, wherein the at least one processor is further configured to provide a second sematic preserving filter to remove side-lobes prior to generation of the radar image.

[0350] Example 134. The least one non-transitory storage media of Example 133, wherein the side-lobes are in one or both elevation and azimuth dimensions.

[0351] Example 135. The least one non-transitory storage media of any one of Examples 107-134, wherein the at least one processor is configured to provide a spatially varying range processing module configured to process the radar data received from the radar unit to determine a range, and wherein the processing parameter comprises a range processing parameter.

[0352] Example 136. The least one non-transitory storage media of Example 135, wherein the at least one processor is configured to provide a frequency varying Doppler processing module configured to process an output of the spatially varying range processing module to determine a velocity, and wherein varying the processing parameter comprises a Doppler processing parameter.

[0353] Example 137. The least one non-transitory storage media of any one of Examples 135 and 136, wherein the processing parameter comprises a window function or a bin size.

[0354] Example 138. The least one non-transitory storage media of any one of Examples 135 and 136, wherein the processing parameter comprises a selected group of bins to which a window function is applied.

[0355] Example 139. The least one non-transitory storage media of any one of Examples 107-138, wherein the at least one processor is further configured to provide a super-resolution angle finding module configured to determine an angle of a radarMOTN.153WO PATENTscatterer with respect to the radar unit, by processing output of a multiple input multiple output (MIMO) processing module provided by the processor

[0356] Example 140. The least one non-transitory storage media of Example 139, wherein the super-resolution angle finding module comprises a transformer-based machine learning model trained using the outputs of an iterative adaptive approach algorithm to determine the angle.

[0357] Example 141. The radar system of any one of Examples 1-71, wherein the radar system is a radar system of a vehicle and the environment surrounding the vehicle.

[0358] Example 142. The radar system of Example 142, wherein the radar unit and the computing system are positioned at different locations in the vehicle.

[0359] Example 143. The method of any one of Examples 72-106, wherein the radar unit and the computing system are positioned at different locations in a vehicle and the environment surrounding the vehicle.

[0360] Example 144. The least one non-transitory storage media of any one of Examples 106-140, wherein the radar unit and the computing system are positioned at different locations in a vehicle and the environment surrounds the vehicle.

[0361] Example 145. An autonomous vehicle, comprising the radar system of any one of examples 1-71; and an autonomous vehicle compute configured to use output of the radar system to control navigation of the autonomous vehicle.

[0362] Example 146. The radar system of any of one Examples 1-71, wherein varying a processing parameter comprises sweeping the processing parameter from a first specified value to a second specified value to determine individual ones of the at least two different values of the processing parameter.

[0363] Example 147. The method of any one of Examples 72-106, wherein varying a processing parameter comprises sweeping the processing parameter from a first specified value to a second specified value to determine individual ones of the at least two different values of the processing parameter.

[0364] Example 148. The least one non-transitory storage media of any one of Examples 106-140, wherein varying a processing parameter comprises sweeping the processing parameter from a first specified value to a second specified value toMOTN.153WO PATENTdetermine individual ones of the at least two different values of the processing parameter.

[0365] Example 149. A radar system comprising: a radar unit configured to generate radar data in response to receiving radio waves reflected by radar-scatterers in an environment; a computing system in data communication with the radar unit, the computing system comprising a processor configured to execute machine-readable instructions to process the radar data and generate processed radar data comprising at least one of a range and a velocity of the radar-scatterers by using at least two different processing parameters for each of the at least one of the range and the velocity.

[0366] Example 150. The radar system of Example 149, wherein at least two different processing parameters comprise at least two different types of parameters.

[0367] Example 151. The radar system of Example 150, wherein at least two different processing parameters comprise at least two different sets of parameters.Terminology

[0368] In this description numerous specific details are set forth in order to provide a thorough understanding of the present disclosure for the purposes of explanation. It will be apparent, however, that the embodiments described by the present disclosure can be practiced without these specific details. In some instances, well-known structures and devices are illustrated in block diagram form in order to avoid unnecessarily obscuring aspects of the present disclosure.

[0369] Specific arrangements or orderings of schematic elements, such as those representing systems, devices, modules, instruction blocks, data elements, and / or the like are illustrated in the drawings for ease of description. However, it will be understood by those skilled in the art that the specific ordering or arrangement of the schematic elements in the drawings is not meant to imply that a particular order or sequence of processing, or separation of processes, is required unless explicitly described as such. Further, the inclusion of a schematic element in a drawing is not meant to imply that such element is required in all embodiments or that the features represented by such elementMOTN.153WO PATENTcannot be included in or combined with other elements in some embodiments unless explicitly described as such.

[0370] Although the terms first, second, third, and / or the like are used to describe various elements, these elements should not be limited by these terms. The terms first, second, third, and / or the like are used only to distinguish one element from another. For example, a first contact could be termed a second contact and, similarly, a second contact could be termed a first contact without departing from the scope of the described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.

[0371] The terminology used in the description of the various described embodiments herein is included for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a,” “an” and “the” are intended to include the plural forms as well and can be used interchangeably with “one or more” or “at least one,” unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “includes,” and / or “comprising,” when used in this description specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0372] As used herein, the term “if” is, optionally, construed to mean “when”, “upon”, “in response to determining,” “in response to detecting,” and / or the like, depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining,” “in response to determining,” “upon detecting [the stated condition or event],” “in response to detecting [the stated condition or event],” and / or the like, depending on the context. Also, as used herein, the terms “has”, “have”, “having”, or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise.

Claims

MOTN.153WO PATENTWHAT IS CLAIMED IS:

1. A radar system comprising:a radar unit configured to generate radar data in response to receiving radio waves reflected by radar-scatterers in an environment;a computing system in data communication with the radar unit, the computing system comprising a processor configured to execute machine-readable instructions to process the radar data and generate processed radar data comprising at least one of a range and a velocity of the radar-scatterers by varying a processing parameter to determine at least two different values of the processing parameter for each of the at least one of the range and the velocity.

2. The radar system of claim 1 , wherein computing system is configured to use each one of the at least two different values of processing parameters to determine the range of one of at least two radar-scatterers at two different positions in the environment.

3. The radar system of claim 1, wherein computing system is configured to use each one of the at least two different values of processing parameters to determine the velocity of one of at least two radar-scatterers at two different positions in the environment.

4. The radar system of claim 1 , wherein the radar data is generated by at least two antennas of the radar unit in response to emission at least two different radiation patterns by the radar unit.

5. The radar system of claim 1 , wherein the data communication of the radar unit and the computing unit is implemented at least in part by a serial data link, and the radar unit comprises a serializer that serializes the radar data for transition through the serial data link.

6. The radar system of claim 5, wherein the data link is configured to transmit data at a rate equal or greater than 3 Gigabit per second.

7. The radar system of claim 1 , wherein the computing system is separated from the radar unit and is configured to receive the radar data from the radar unit via a cable.

8. The radar system of claim 1 , wherein the radar unit does not use a digital signal processing circuit to generate radar data.MOTN.153WO PATENT9. The radar system of claim 1 , wherein the processor uses a trained machine learning model to vary the processing parameter.

10. The radar system of claim 1 , wherein radar data comprises digitized data output by an analog-to-digital converter of the radar unit in response to receiving reflected radar signals from a transceiver of the radar unit.

11. The radar system of claim 1 , wherein the processor further executes machine readable instructions to provide a first sematic preserving filter configured to filter the processed radar data to suppress one or both noise and side-lobe data, wherein the sidelobe data comprises side-lobes in one or both range and velocity dimensions.

12. The radar system of claim 11, wherein first sematic preserving filter is configured to substantially preserve a shape parameter of a radar-scatterer in a radar image.

13. The radar system of claim 1 , wherein the radar-scatterers include at least one target object and at least one background, and the radar data include target data generated by the target object and clutter data generated by the background.

14. The radar system of claim 13, wherein the computing system is configured to retain at least a portion of clutter data when processing the target data and to generate the processed radar data using a machine learning model based on both target data and clutter data.

15. The radar system of claim 1, wherein the processor further executes machine readable instructions to provide a second sematic preserving filter to remove side-lobes prior to generation of the radar image, and wherein the side-lobes are in one or both elevation and azimuth dimensions.

16. The radar system of claim 1, wherein the processor executes the machine-readable instructions to provide a spatially varying range processing module configured to process the radar data received from the radar unit to determine a range, and wherein the processing parameter comprises a range processing parameter.

17. The radar system of claim 16, wherein the processor executes machine readable instructions to provide a frequency varying Doppler processing module configured to process an output of the spatially varying range processing module to determine a velocity, and wherein varying the processing parameter comprises a Doppler processing parameter.MOTN.153WO PATENT18. The radar system of any one of claims 16 and 17, wherein the processing parameter comprises a window function, a bin size, or a selected group of bins to which a window function is applied.

19. The radar system of claim 1 , wherein the processor executes machine readable instructions to provide a super-resolution angle finding module configured to determine an angle of a radar scatterer with respect to the radar unit, by processing output of a multiple input multiple output (MIMO) processing module provided by the processor.

20. The radar system of claim 19, wherein the super-resolution angle finding module comprises a transformer-based machine learning model trained using the outputs of an iterative adaptive approach algorithm to determine the angle.

Images