Systems and methods for providing notifications of detected objects in a camera monitor system

Abstract

A method for a camera monitor system (CMS) of a commercial vehicle includes detecting that one or more warning criteria for notifying a driver about an object near the vehicle have been satisfied, the detecting including image analysis of an image recorded by a CMS camera. The method further includes determining a relative location of the object with respect to the vehicle and, in response to the detecting, providing a notification to the driver indicating the object's relative location. The notification includes displaying a graphical warning indication on a CMS electronic display at a location determined from the object's relative location and within a region of the display selected to comply with a legally prescribed field-of- view requirement.

Description

SYSTEMS AND METHODS FOR PROVIDING NOTIFICATIONS OF DETECTED OBJECTS IN A CAMERA MONITOR SYSTEMCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No.63 / 721,645, filed November 18, 2024, the disclosures of which are incorporated herein by reference in their entirety.TECHNICAL FIELD

[0002] This disclosure relates to camera monitor systems (CMS), and more particularly systems and methods for notifying a driver of nearby objects, such as things that may lead to vehicle damage (e.g., animals, road obstacles), and / or bystanders who could potentially be injured by the driver.BACKGROUND

[0003] Vehicle camera systems for mirror replacement or for supplementing mirror views are utilized in commercial vehicles to enhance the ability of a vehicle operator to see a surrounding environment of the commercial vehicle. These systems are known as “camera monitor systems” (CMS), and they utilize one or more cameras mounted to a tractor of a commercial vehicle to provide an enhanced field of view to a vehicle operator of an area surrounding a trailer of the commercial vehicle. CMS may also include cameras in locations not typically associated with a mirror, such as a rear camera (e.g., a trailer camera) that records images of an area behind a vehicle, a camera that records an area in front of a vehicle, etc.

[0004] Some vehicles also include object detection sensors, such as ultrasonic, LIDAR, and / or RADAR sensors.SUMMARY

[0005] In implementations, a method for a camera monitor system (CMS) of a commercial vehicle includes detecting that one or more warning criteria for notifying a driver about an object near the vehicle have been satisfied, the detecting including image analysis of an image recorded by a CMS camera. The method further includes determining a relative location of the object with respect to the vehicle and, in response to the detecting, providing a notification to the driver indicating the object’s relative location. The notification includesdisplaying a graphical warning indication on a CMS electronic display at a location determined from the object’s relative location and within a region of the display selected to comply with a legally prescribed field-of-view requirement.

[0006] In implementations, the notification may include generating an auditory warning emitted from a vehicle sound emitter on a side corresponding to the object’s relative location.

[0007] In implementations, the auditory warning may be spatialized between left and right cabin speakers to represent a lateral position of the object.

[0008] In implementations, the notification may include generating haptic feedback to the driver through a haptic feedback device. The haptic feedback may be modulated based on the object’s relative location.

[0009] In implementations, the haptic feedback may be selectively generated in a seat region corresponding to the side of the vehicle at which the object is located.

[0010] In implementations, the notification may include generating at least one supplemental alert selected from: (i) an auditory warning emitted from a vehicle sound emitter on a side corresponding to the object’s relative location; and (ii) haptic feedback to the driver through a haptic feedback device, the haptic feedback modulated based on the object’s relative location.

[0011] In implementations, the method may include dynamically adjusting a position or an opacity of the graphical warning indication based on real-time image content to maintain compliance with field-of-view restrictions.

[0012] In implementations, the graphical warning indication may be displayed in an upper-left corner of the electronic display corresponding to a detected object located along a driver side of the vehicle.

[0013] In implementations, the location of the graphical warning indication may be selected to avoid overlap with image regions depicting a trailer or road surface within the camera view.

[0014] In implementations, the graphical warning indication may be displayed in an upper-right corner of the electronic display corresponding to a detected object located along a passenger side of the vehicle.

[0015] In implementations, detecting that the warning criteria have been satisfied may include receiving a signal indicative of the object from a source external to the vehicle, thesignal being a vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), or vehicle-to-infrastructure (V2I) communication.

[0016] In implementations, detecting the object may include processing data from at least one object-detection sensor selected from an ultrasonic sensor, a LIDAR sensor, or a RADAR sensor.

[0017] In implementations, the graphical warning indication may include a first graphic representing the vehicle and a second graphic representing the object. The second graphic may be positioned relative to the first graphic to indicate the object’s relative location.

[0018] In implementations, the graphical warning indication may be displayed in a corner region of the electronic display corresponding to a direction of the detected object relative to the vehicle.

[0019] In implementations, providing the notification may include concurrently displaying the graphical warning indication, emitting an auditory warning from a vehicle sound emitter on a side corresponding to the object’s relative location, and generating haptic feedback through a haptic feedback device. The camera monitor system may dynamically adjust at least one of a position or an opacity of the graphical warning indication based on real-time image content to maintain compliance with a legally prescribed field-of-view requirement.

[0020] In implementations, the graphical warning indication may be displayed at a location on the electronic display corresponding to a lateral position of the detected object, and an auditory warning may be simultaneously emitted with a lateral balance between left and right cabin speakers representing the same lateral position.

[0021] In implementations, when the detected object is located along a driver side of the vehicle, the camera monitor system may concurrently display the graphical warning indication in an upper-left corner of a left-side display, may emit an auditory warning from a left-side cabin speaker, and may activate haptic feedback in a left-side region of a driver seat.

[0022] In implementations, objects detected through image analysis may be indicated using a first graphical icon, and objects detected via vehicle-to-vehicle or vehicle-to-infrastructure communication may be indicated using a second, visually distinct icon or color.

[0023] In implementations, the notification may include at least one supplemental alert selected from: (i) an auditory warning emitted from a vehicle sound emitter on a side corresponding to the object’s relative location; and (ii) haptic feedback to the driver through ahaptic feedback device, the haptic feedback modulated based on the object’s relative location. The method may also include dynamically adjusting a position or an opacity of the graphical warning indication based on real-time image content to maintain field-of-view compliance, with the location of the graphical warning indication selected to avoid overlap with image regions depicting a trailer or road surface within the camera view. Detecting the object may include processing data from at least one object-detection sensor selected from an ultrasonic sensor, a LIDAR sensor, or a RADAR sensor.

[0024] In implementations, the notification may include at least one supplemental alert selected from: (i) an auditory warning emitted from a vehicle sound emitter on a side corresponding to the object’s relative location; and (ii) haptic feedback to the driver through a haptic feedback device, the haptic feedback modulated based on the object’s relative location. The method may also include dynamically adjusting a position or an opacity of the graphical warning indication based on real-time image content to maintain field-of-view compliance, with the location of the graphical warning indication selected to avoid overlap with image regions depicting a trailer or road surface within the camera view. Detecting that the warning criteria have been satisfied may include receiving a signal indicative of the object from a source external to the vehicle, such as a V2V, V2P, or V2I communication.

[0025] In implementations, a camera monitor system (CMS) for a vehicle including a tractor and a trailer includes a camera having an image capture unit configured to provide a field of view of the vehicle, a display configured to depict at least a portion of the field of view, and processing circuitry operatively connected to memory. The processing circuitry is configured to detect that one or more warning criteria for notifying a driver about an object near the vehicle have been satisfied, the detecting including image analysis of an image recorded by a CMS camera. The processing circuitry determines a relative location of the object with respect to the vehicle and, in response to the detecting, provides a notification to the driver indicating the object’s relative location. The notification includes displaying a graphical warning indication on the display at a location determined from the object’s relative location and within a region of the display selected to comply with a legally prescribed field-of-view requirement.

[0026] In implementations, the processing circuitry may be further configured to generate an auditory warning through a sound emitter located on a side of the vehicle corresponding to the relative location of the detected object.

[0027] In implementations, the auditory warning may be spatialized between left and right cabin speakers to represent a lateral position of the detected object relative to a longitudinal centerline of the vehicle, thereby creating a spatial perception of the object’s location.

[0028] In implementations, the processing circuitry may be further configured to generate haptic feedback through a haptic feedback device embedded in a driver seat or display. The haptic feedback may be modulated based on the relative location or severity of the detected object.

[0029] In implementations, the processing circuitry may dynamically adjust one or more of a position, brightness, or opacity of the graphical warning indication based on realtime image content of the display to maintain compliance with a legally prescribed field-of-view requirement.

[0030] In implementations, the processing circuitry may display the graphical warning indication in an upper-left corner of the display corresponding to a detected object located along a driver side of the vehicle.

[0031] In implementations, the processing circuitry may select the location of the graphical warning indication to avoid overlap with image regions depicting a trailer or road surface within the field of view of the camera, thereby minimizing obstruction of critical visual content.

[0032] In implementations, the processing circuitry may display the graphical warning indication in an upper-right corner of the display corresponding to a detected object located along a passenger side of the vehicle.

[0033] In implementations, the processing circuitry may determine that the warning criteria have been satisfied based on a signal received from a source external to the vehicle, the signal including at least one of a vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), or vehicle-to-infrastructure (V2I) communication.

[0034] In implementations, the processing circuitry may detect the object based on data from at least one object detection sensor selected from an ultrasonic sensor, a LIDAR sensor, or a RADAR sensor, and may fuse such sensor data with image analysis to enhance detection reliability.

[0035] In implementations, the processing circuitry may generate the graphical warning indication including a first graphic representing the vehicle and a second graphicrepresenting the detected object. The second graphic may be positioned relative to the first graphic to indicate the relative location of the object.

[0036] These and other features of the present invention can be best understood from the following specification and drawings, the following of which is a brief description.BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The disclosure can be further understood by reference to the following detailed description when considered in connection with the accompanying drawings wherein:

[0038] Figure 1 is a schematic front view of a commercial truck with a camera mirror system (CMS) used to provide at least Class II and Class IV views.

[0039] Figure 2 is a schematic birds-eye view of the commercial truck of Figure 1 with a CMS providing Class II, Class IV, Class V, Class VI, and Class VIII views.

[0040] Figure 3 is a schematic top view of an example vehicle cabin interior.

[0041] Figure 4 is a perspective view of the vehicle cabin interior of Figure 3.

[0042] Figures 5A-5E schematically illustrate example graphical warning notifications.

[0043] Figures 6A, 6B, 7A, 7B, 8A, 8B, 9, and 10 each depict example graphical warning display configurations for various CMS displays.

[0044] The embodiments, examples, and alternatives of the claims, and / or the following description and drawings, including any of their various aspects or respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, unless such features are incompatible.DETAILED DESCRIPTION

[0045] Schematic views of a commercial vehicle 10 are illustrated in Figures 1-4. The commercial vehicle 10 includes a vehicle cab or “tractor” 12 for pulling a trailer 14, where the trailer 14 pivots with respect to the tractor 12 during turns. Although the commercial vehicle 10 is depicted as a commercial truck with a single trailer in this disclosure, it is understood that other commercial vehicle configurations may be used (e.g., different types or quantities of trailers).

[0046] A pair of camera arms 16A-B include a respective base that is secured to, for example, the tractor 12. A pivoting arm is supported by the base and may articulate relative thereto. At least one rearward facing camera 20A-B is arranged respectively on or within thecamera arms 16A-B. The exterior cameras 20A-B respectively provide an exterior field of view FOVEXI, FOVEX2 that each include at least one of Class II and Class IV views (Fig. 2), which are legally prescribed views in the commercial trucking industry.

[0047] The Class II view on a given side of the commercial vehicle 10 is a subset of the class IV view of the same side of the commercial vehicle 10. Multiple cameras also may be used in each camera arm 16A-B to provide these views, if desired. Class II (narrow) and Class IV (wide angle) views are defined in European R46 legislation, for example, and the United States and other countries have similar drive visibility requirements for commercial trucks. Any reference to a “Class” view is not intended to be limiting, but is intended as an example of the type of view provided to a display from a particular camera.

[0048] Each camera arm 16A-16B may also provide a housing that encloses electronics, e.g., a controller, that are configured to provide various features of the CMS 15. The camera arms 16A-B may be mounted either at a roof-mount location over the cab door (as shown), or on a door-mounted bracket or station, for example.

[0049] If video of Class V and / or Class VI views is also desired, a camera housing 16C and camera 20C may be arranged at or near the front of the commercial vehicle 10 to provide those views (Figure 2).

[0050] A backup camera 20D may be provided which provides a field of view FOVEXS. The backup camera 20D may be mounted at a top / centerline of the trailer, at a bumper / bed level of the trailer, or at a top-corner of the back of the trailer, for example. Alternatively, or in addition to the rear trailer camera, a “fifth wheel camera” 20E may be provided that is mounted to a rear of the tractor 12 and that provides a field of view FOVEX4. The fifth wheel camera 20E may be mounted anywhere between the lateral plane of the fifth wheel fixture and the top / roof edge of the tractor, for example.

[0051] Figure 3 is a schematic top view of an example vehicle cabin interior 24, and Figure 4 is a perspective view of the vehicle cabin interior 24. Referring now to Figures 3-4 with continued reference to Figures 1-2, electronic displays 18A-E (e.g., which may be video displays, such as LCD displays) and cameras 20A-E are shown. The various electronic displays 18A-E and cameras 20A-E are part of a camera monitor system (CMS) 15, and therefore act as CMS displays and CMS cameras. As used herein, a “CMS camera” 20 is a camera configured to record images of an environment surrounding a commercial vehicle 10, and a “CMS display” 18 is an electronic display (e.g., an LCD) that is configured to image feeds from those cameras.

[0052] The CMS 15 includes a CMS electronic control unit (ECU) 22 that acts as a controller and includes processing circuitry that supports operation of the CMS 15. The CMSECU 22 is operatively connected to memory (which may include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or nonvolatile memory elements (e.g., ROM, hard drive, tape, CD-ROM, etc.). The processing circuitry may include one or more microprocessors, microcontrollers, application specific integrated circuits (ASICs), or the like.

[0053] The CMS ECU 22 is in communication with the various CMS cameras 20A-E and also optionally also one or more of: at least one object detection sensor 40, at least one sound emitter 42, and at least one wireless signaling device 44 (e.g., a wireless transceiver) .

[0054] The object detection sensor 40 may include an ultrasonic sensor, LIDAR sensor, or RADAR sensor, for example, and a plurality of the object detection sensors 40 may be distributed along a perimeter of the commercial vehicle to detect nearby objects. In implementations, the ECU 22 may combine detections from multiple sensor types, such as to confirm object presence and / or reduce false positives.

[0055] The sound emitter 42 may be a speaker or buzzer, and may correspond to a vehicle cabin speaker (e.g., typically used for radio I music playback) or to a speaker built-into I collocated with one or more of the displays 18, for example.

[0056] The wireless signaling device 44 is configured to communicate with external devices to receive and / or transmit vehicle-to-vehicle (V2V) signals, vehicle-to-pedestrian (V2P) signals, and / or vehicle-to-infrastructure (V2I) signals.

[0057] The CMS displays 18A-B are arranged on each of the driver and passenger sides within the vehicle cab 12 on or near the A-pillars 19A-B to display Class II and Class IV views on its respective side of the commercial vehicle 10, which provide rear facing side views along the commercial vehicle 10 that are captured by the exterior cameras 20A-B.

[0058] As discussed above, if video of Class V and Class VI views are also desired, the camera housing 16C and camera 20C may be arranged at or near the front of the commercial vehicle 10 to provide those views (Figure 2). In the example of Figure 3, additional displays 18C-E are provided. Display 18C is arranged in the vehicle cabin interior 24 near the top center of the windshield may be used to display the Class V and Class VI views, which are toward the front of the commercial vehicle 10, or a backup camera view (from camera 20D or 20E) to the driver. Display 18D is provided in a center console area of the vehicle cabin interior 24, and may be used for other purposes, such as navigation, infotainment, etc. Display 18E may be part of an instrument cluster, for example.

[0059] If desired, the camera arms 16A-B may include conventional mirrors integrated with them as well, although the CMS 15 may be used to entirely replace mirrors. Inadditional examples, each side can include multiple camera arms, with each arm housing one or more cameras and / or mirrors.

[0060] The vehicle cabin interior 24 includes two seats 48A-B, and the driver seat 48A may include a haptic feedback device 50 (e.g., a haptic motor) for generating vibration. In implementations, a haptic motor may also be provided in one of the CMS displays 18. The haptic motor(s) are on communication with, and are controlled by, the ECU 22.

[0061] The CMS 15 is configured to detect whether one or more warning criteria for notifying a driver of the commercial vehicle 10 about a nearby object have been satisfied. The warnings may be provided, e.g., for detected pedestrians, bicyclists, animals, passenger cars, commercial vehicles, and / or road hazards, and serve to notify the driver of nearby objects which may lead to vehicle damage and / or injury to bystanders. The detection may be performed using techniques based on analysis of images recorded by the cameras 20, and / or based on data from the object detection sensor(s) 40. Alternatively or in addition, the detection may be based on a V2V, V2P, and / or V2I signals received via the wireless signaling device 44. In one or more embodiments, the detection includes detecting the type of object (e.g., human or other vehicle), and may additionally or alternatively indicate a location of the object relative to the vehicle 10.

[0062] In some embodiments, the CMS 15 may fuse data from multiple sensing modalities (e.g., visual, radar, ultrasonic, and wireless communications) to improve detection accuracy and reduce false positives. The system may also assign a confidence value to each detected object based on signal quality, object persistence across frames, and agreement between sensors.

[0063] Upon determining that one or more warning criteria are satisfied, the CMS 15 may generate a driver alert through one or more channels. The type, intensity, or duration of the alert may vary based on a severity level of the detected event (e.g., proximity, relative velocity, or predicted collision risk). In implementations, based on the detection, the CMS 15 provides a notification to the driver, which includes one or more of:a graphical warning indication on an electronic display 18 of the CMS 15;an auditory warning indication provided through the vehicle sound emitter(s) 42; and haptic feedback to a driver of the commercial vehicle through the haptic feedback device 50 embedded in the seat 48A of the vehicle 10, or embedded in a display 18 of the commercial vehicle.

[0064] In implementations, the detection includes identifying the type of object (e.g., pedestrian, vehicle, or static obstacle) and determining its position and motion relative to the vehicle 10.

[0065] The detection of nearby objects by the CMS 15 may utilize various image analysis and sensor processing techniques executed by the ECU 22 to interpret data from the CMS cameras 20A-E and, in some implementations, from the object detection sensor(s) 40. The detection of nearby objects by the CMS 15 may utilize image analysis techniques, such as deep learning algorithms (e.g., convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformer-based vision models, or other suitable machine learning or rulebased techniques for object detection) executed by the CMS ECU 22, to process images from CMS cameras 20A-E. For example, a deep learning model trained on a dataset of pedestrians, vehicles, and road hazards may classify objects and determine their bounding box coordinates within the camera’s field of view.

[0066] Additionally or alternatively, the object detection sensor(s) 40 may include, for example, LIDAR, RADAR, ultrasonic, or infrared sensors that generate range or point cloud data representing objects in the vehicle’s surroundings. The ECU 22 may process these data using signal filtering, clustering, or feature extraction techniques to verify object presence, estimate object size and distance, and reduce false positives.

[0067] In some embodiments, the ECU 22 may perform sensor fusion by combining data from multiple cameras and / or sensor types to enhance detection accuracy, improve object localization, and maintain reliable performance under varying environmental conditions (e.g., low light, glare, rain, or fog). The system may dynamically select or weight inputs from different sensors.

[0068] Figures 5A-E schematically illustrate example graphical warning notifications 54A-E, with notifications 54A-D corresponding to human detected objects, and notification 54E corresponding to a vehicle detected object. In each notification 54, a birdseye vehicle depiction 56 is provided, and a depiction of the detected object is also provided (a human 58 in Figures 5A-D and a vehicle 60 in Figure 5E). The size, color, shape, or animation of the notification may vary depending on a level of alert, system mode, or proximity of the detected object.

[0069] In each of the example notifications of Figures 5A-E, the location of the detected object in the notification 54 corresponds to a relative location of the detected object with respect to the vehicle commercial vehicle. Thus, in notification 54A, the detected human is located to the front / left of the vehicle. In notification 54B, the detected human is located to the front / right of the vehicle. In notification 54C, the detected human is located to the rear / left of the vehicle. In notification 54D, the detected human is located to the rear / right of the vehicle. Similarly, in notification 54E, the vehicle 60 is located to the rear / right of the vehicle. The graphical notifications may employ different colors, shading patterns, blinking rates, ortransparency levels to indicate warning severity or confidence levels. In certain embodiments, auditory or haptic cues may be issued concurrently with the visual notification to reinforce the driver alert.

[0070] Figure 6A, 6B, 7A, 7B, 8A, 8B, 9, and 10 each depict example graphical warning display configurations for various CMS displays 18. The graphical notifications may be displayed as overlays, inset windows, icons, or peripheral indicators depending on the available screen space, detected event type, or user-selected interface mode.

[0071] Figure 6A depicts an example configuration for display 18A, which depicts a Class II view (upper display area) and Class IV view (lower display area). The notification 54D is provided in an upper left corner of the Class II area, such that the notification 54D does not overlap the trailer 14. Positioning the notification in this manner is advantageous because that region is least likely to depict the detected object itself, thereby minimizing obstruction of relevant image content. In other embodiments, the placement of notifications may be dynamically adjusted based on the detected object’s location, driver preferences, or contextual parameters (e.g., vehicle speed, active camera mode, ambient noise, or ambient lighting conditions) to maintain visibility without distracting the driver.

[0072] Figure 6B depicts an example configuration for display 18B, which depicts a Class II view (upper display area) and Class IV view (lower display area). The notification 54E is provided in an upper right corner of the Class II area, such that the notification 54E does not overlap the trailer 14. Positioning the notification in this location is advantageous because the upper right corner is typically the least likely portion of the display to include the detected object itself, thereby preserving the visibility of relevant camera imagery. In other embodiments, the notification placement may be dynamically determined based on object location data, user interface design parameters, or vehicle configuration, ensuring that warnings are displayed in an area that maximizes visibility while minimizing distraction.

[0073] Figure 7A depicts another example configuration for display 18A. In this example, the notification 54 is provided between the Class II (upper area) and Class IV (lower area), along with a written portion (the word “Caution”). The written portion may supplement or replace a graphical icon, and may vary depending on the type or severity of the detected object or event. In some embodiments, the textual and graphical portions may be combined, color-coded, or animated to draw the driver’s attention, or may be accompanied by audio or haptic alerts generated by the ECU 22 to further reinforce the warning.

[0074] Figure 7B depicts another example configuration for display 18B. Here too, the notification 54 is provided between the Class II (upper area) and Class IV (lower area), along with a written portion (the word “Caution”). The text may be displayed in combinationwith or in place of a graphical icon, and may appear in varying colors, sizes, or font styles to emphasize urgency or alert type. Similar to other configurations, the location of the notification may be selected to avoid obstructing key portions of the live video feed, while maintaining clear visibility to the driver. In implementations, the notification area may expand, flash, or change color based on the detected object’s proximity or movement, or may present different messages corresponding to distinct warning thresholds.

[0075] Figure 8A depicts an example configuration for a Class V camera on a driver side of the commercial vehicle 10, in which the notification 54 is provided in an upper right corner. This placement helps ensure that the alert remains visible to the driver while minimizing interference with the primary image region most likely to display adjacent traffic or nearby obstacles. In alternative embodiments, the notification 54 may be positioned in another area of the display based on object location, user preference, or system configuration.

[0076] Figure 8B depicts an example configuration for a Class V camera on a passenger side of the commercial vehicle 10, in which the notification 54 is provided in an upper left corner. The mirrored positioning between the driver- and passenger-side Class V displays allows for consistent alert visibility and user orientation, enabling the driver to intuitively associate the side of the vehicle with the side of the alert. The notification placement may also be adjusted automatically by the ECU 22 to account for camera calibration, image cropping, or display layout changes.

[0077] Figure 9 depicts an example configuration for a class VI camera (e.g., camera 20C), in which the notification 54 is provided in a top right corner. Placement of the notification in the upper corner allows the driver to view alerts without obstructing nearby road surface imagery, which may be important for detecting pedestrians or small objects near the frontof the vehicle. In other implementations, multiple notifications may appear simultaneously if objects are detected in multiple regions within the Class VI field of view.

[0078] Figure 10 depicts an example configuration for the backup camera 20D, in which the notification 54 is provided in a top left corner. This positioning maintains visibility while avoiding overlap with visual cues shown in rear-view displays. In some embodiments, the backup camera display may include dynamic overlay regions where notification placement is automatically adjusted based on vehicle steering angle, trailer articulation, or object trajectory, ensuring the alert remains visible and contextually aligned with the detected hazard.

[0079] The configurations shown in Figures 5A-10 illustrate example implementations of graphical warning notifications across different CMS display views and camera classes. A skilled person having the benefit of this disclosure will appreciate that the positioning, size, and format of each notification 54 may vary depending on factors such ascamera orientation, detected object location, display layout, and driver interface preferences. In some embodiments, the ECU 22 may dynamically determine or adjust the notification placement based on real-time vehicle conditions, display mode, vehicle sensor readings, and / or the driver’s line of sight to optimize visibility and minimize distraction. Furthermore, notifications may be displayed concurrently on multiple CMS displays 18, or transferred between displays as the detected object moves around the vehicle.

[0080] In implementations, audio notifications may be provided in a directional manner to correspond with the detected object’s location relative to the commercial vehicle 10. For example, an audio alert may be emitted from the vicinity of the display 18A and / or a left-side cabin speaker when an object is detected along the driver side of the vehicle, and from the vicinity of the display 18B and / or a right-side cabin speaker when an object is detected along the passenger side. Characteristics such as intensity, tone, or pattern of the audio signal may further vary based on characteristics of the object, such as object proximity, movement, or risk level, for example. Additionally, the audio notifications may be synchronized with corresponding visual or haptic alerts to reinforce the warning and enhance driver awareness.

[0081] In one or more embodiments, haptic feedback is provided through the haptic feedback device 50 in conjunction with detected object notifications. The haptic feedback may be generated for all detected objects or selectively for those determined to present a higher level of risk or urgency (e.g., objects in close proximity or on a collision trajectory). The ECU 22 may vary the intensity, frequency, duration, or pattern of vibration to convey different alert levels or object types, enabling the driver to distinguish between various hazard conditions without relying solely on visual or auditory cues. The use of haptic feedback may be synchronized with corresponding visual and / or audio alerts to reinforce the driver warning through multiple sensory channels, thereby enhancing overall situational awareness.

[0082] Auditory cues may be utilized to indicate the direction or position of a detected object. In implementations, directional auditory cues are achieved by modulating relative speaker output between left and right cabin speakers to create a spatialized perception of the object’s location. For example, a detected hazard along the driver side may produce a left-weighted alert tone, while a hazard along the passenger side may produce a right-weighted tone. Alternatively, speakers integrated into or near the CMS displays 18A-18B may emit localized warning tones corresponding to the display on which the notification is presented. In implementations, the warning tone may be non-directional and broadcast uniformly across all speakers.

[0083] In implementations, the camera monitor system (CMS) 15 may determine a lateral position of a detected object relative to the longitudinal centerline of the commercialvehicle 10. The lateral position may be expressed as a direction (e.g., left or right side) or as a relative distance offset within the camera field of view or vehicle coordinate system. The CMS 15 may utilize this lateral position information to influence one or more alert modalities. For example, an auditory warning may be output with a lateral balance or spatial weighting between left and right cabin speakers corresponding to the detected object’s lateral position. Similarly, the graphical warning indication may be displayed at a location on the electronic display corresponding to the detected object’s lateral position, and haptic feedback may be provided on a corresponding side of the driver seat.

[0084] Haptic feedback may also be provided to the driver to supplement or replace visual and auditory warnings. In implementations, a vibration motor may be embedded within the driver seat 48A to provide tactile cues. The ECU 22 may selectively activate a left or right seat motor corresponding to the location of the detected hazard or vary the vibration intensity based on the severity of the alert. In another embodiment, a haptic actuator is integrated into one or more of the CMS displays 18, such that the screen surface provides a brief tactile pulse coinciding with the visual icon. In implementations, haptic feedback may additionally or alternatively be incorporated into other vehicle subsystems or accessories — such as the steering wheel — to provide directional or event-specific feedback directly through the driver’s primary point of control.

[0085] The graphical notifications 54 may correspond to and / or comport with ECE R159 and / or ISO 7000, for example.

[0086] As discussed above, the detected object notification presented to the driver may be visual, auditory, haptic, or any combination thereof. The multimodal nature of the notifications allows the system to adapt to environmental conditions and / or driver preferences, enhancing the likelihood that the alert will be perceived and appropriately acted upon.

[0087] The graphical notifications discussed herein provide a way for display notification without traditional bounding boxes, which may not be feasible for systems deployed in Europe due to ECE regulations such as ECE R46 which prevents overlays obstructing more than 2.5% of the legally prescribed field-of-view for a given camera view region and must be temporary in nature. Similarly in the United States SAE J3155 prevents overlays obstructing more than 5% of the legally prescribed field-of-view for a given camera view region.

[0088] The ECU 22 may dynamically adjust one or more of the position, transparency, brightness, or size of graphical warning icons so that overlays do not exceed allowable field-of-view coverage. For example, when multiple warnings are activesimultaneously, the ECU may prioritize placement in unoccupied display corners or temporarily reduce icon opacity to maintain compliance while preserving visibility to the driver.

[0089] In implementations, the ECU 22 performs real-time analysis of the display content to determine appropriate placement of graphical warning notifications 54. For instance, the ECU 22 may execute an algorithm that identifies display regions corresponding to primary visual content — such as the trailer 14, vehicle outlines, or road surface — and dynamically positions notifications in less visually prominent areas, such as unoccupied display corners. The ECU 22 may further adapt the appearance and persistence of notifications based on operational context (e.g., vehicle speed, ambient lighting, or alert priority), such that the warnings remain visible, compliant, and minimally intrusive to the driver’s field of view.

[0090] Through these adaptive display and alert-management techniques, the CMS 15 allows driver notifications to be effective under a wide range of vehicle and environmental conditions while maintaining compliance with applicable visibility regulations. By coordinating visual, auditory, and haptic warning outputs, the ECU 22 provides context-appropriate alerts that enhance driver situational awareness without introducing excessive distraction or visual clutter. These capabilities allow the CMS 15 to deliver timely and intelligible warnings that scale dynamically with one or more of object proximity, system confidence, and vehicle operating state, which may improve safety and / or user experience in implementations.

[0091] The present disclosure provides methods for delivering easily identifiable and self-explanatory graphical notifications (e.g., icons) that communicate hazard information to the driver while remaining compliant with regulatory field-of-view restrictions, such as those defined in ECE R46 and SAE J3155. The notifications are configured to convey warnings clearly and intuitively without requiring extensive driver training, thereby enhancing safety and reducing the likelihood of bodily injury and / or vehicle collisions.

[0092] The graphical icons may conform to established human-machine interface standards, such as ISO 7000 or ECE R159. The icons may employ specific color schemes, such as yellow for cautionary alerts and red for urgent or collision-imminent warnings, which may minimize the need for driver training and / or promote immediate recognition across vehicle types and regions.

[0093] V2x (i.e., V2V, V2I, and / or V2P) signals may be utilized as an object detection technique. Notifications generated in response to such signals may be visually or audibly differentiated from those based on direct sensor detections, thereby enabling the driverto recognize the origin of the alert and understand whether it was derived from on-board sensing or from an external V2X transmission.

[0094] Although a commercial vehicle 10 is discussed at length above, it is understood that the techniques discussed herein could be applied to other types of vehicles as well (e.g., passenger cars, offroad equipment (e.g., mining equipment), rigid trucks, and buses).

[0095] To further maintain system reliability and data integrity, the CMS 15 may incorporate data security and cybersecurity features that restrict unauthorized access, interception, or modification of sensor or communication data. Secure communication protocols, encryption of stored and transmitted detection data, and authenticated or digitally signed software updates may be employed to preserve the fidelity of the notifications presented to the driver and to ensure the integrity of the system’s object detection and warning functions.

[0096] The systems and methods described herein provide a flexible framework for notifying a vehicle operator of detected objects in a manner that enhances driver awareness while maintaining compliance with applicable visibility and safety regulations. By enabling coordinated operation among multiple sensing, processing, and alerting components — such as camera-based image analysis, auxiliary object detection sensors, and adaptive display management — the disclosed techniques achieve reliable and intuitive notification behavior under a wide variety of vehicle and environmental conditions. The overall arrangement provides a technical effect of improving the timeliness, clarity, and contextual relevance of driver alerts, while minimizing visual obstruction and driver distraction. These benefits arise from the integrated control logic and adaptive presentation strategies described herein, which collectively offer functionality and performance not attainable through conventional, independently operating CMS or alert systems.

[0097] In some aspects, the techniques described herein relate to a method for a camera monitor system (CMS) of a commercial vehicle, including: detecting that one or more warning criteria for notifying a driver of the commercial vehicle about an object near the vehicle have been satisfied, and the detecting includes image analysis of an image recorded by a camera of the CMS; determining a relative location of the object with respect to the commercial vehicle; and in response to the detecting, providing a notification to the driver indicating a relative location of the object with respect to the vehicle, the notification including: displaying a graphical warning indication on an electronic display of the CMS at a location determined based on the relative location of the object and within a region of the display selected to comply with a legally prescribed field-of-view requirement.

[0098] In some aspects, the techniques described herein relate to a method, the notification further including generating an auditory warning emitted from a vehicle sound emitter on a side corresponding to the relative location of the object.

[0099] In some aspects, the techniques described herein relate to a method, and the auditory warning is spatialized between left and right cabin speakers to represent a lateral position of the object.

[0100] In some aspects, the techniques described herein relate to a method, the notification further including generating haptic feedback to the driver through a haptic feedback device, the haptic feedback modulated based on the relative location of the object.

[0101] In some aspects, the techniques described herein relate to a method, and the haptic feedback is selectively generated in a seat region corresponding to a side of the vehicle of the object.

[0102] In some aspects, the techniques described herein relate to a method, the notification further including: generating at least one of the following supplemental alerts: (i) an auditory warning emitted from a vehicle sound emitter on a side corresponding to the relative location of the object; and (ii) haptic feedback to the driver through a haptic feedback device, the haptic feedback modulated based on the relative location of the object.

[0103] In some aspects, the techniques described herein relate to a method, including dynamically adjusting a position or opacity of the graphical warning indication based on real-time image content to maintain compliance with field-of-view restrictions.

[0104] In some aspects, the techniques described herein relate to a method, and the graphical warning indication is displayed in an upper left corner of the electronic display corresponding to a detected object located along a driver side of the commercial vehicle.

[0105] In some aspects, the techniques described herein relate to a method, and the location of the graphical warning indication is selected to avoid overlap with image regions depicting a trailer or road surface within a view of the camera.

[0106] In some aspects, the techniques described herein relate to a method, and the graphical warning indication is displayed in an upper right corner of the electronic display corresponding to a detected object located along a passenger side of the commercial vehicle.

[0107] In some aspects, the techniques described herein relate to a method, and detecting that the warning criteria have been satisfied includes receiving a signal indicative of the object from a source external to the commercial vehicle, the signal being a vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), or vehicle-to-infrastructure (V2I) communication.

[0108] In some aspects, the techniques described herein relate to a method, and detecting the object includes processing data from at least one object detection sensor selected from an ultrasonic sensor, a LIDAR sensor, or a RADAR sensor.

[0109] In some aspects, the techniques described herein relate to a method, and the graphical warning indication includes a first graphic representing the commercial vehicle and a second graphic representing the object, the second graphic being positioned relative to the first graphic to indicate the relative location of the object.

[0110] In some aspects, the techniques described herein relate to a method, and the graphical warning indication is displayed in a corner region of the electronic display corresponding to a direction of the detected object relative to the commercial vehicle.

[0111] In some aspects, the techniques described herein relate to a method, and providing the notification includes concurrently displaying the graphical warning indication, emitting an auditory warning from a vehicle sound emitter on a side corresponding to the relative location of the object, and generating haptic feedback through a haptic feedback device, the camera monitor system dynamically adjusting at least one of a position or an opacity of the graphical warning indication based on real-time image content to maintain compliance with a legally prescribed field-of-view requirement.

[0112] In some aspects, the techniques described herein relate to a method, and the graphical warning indication is displayed at a location on the electronic display corresponding to a lateral position of the detected object, and an auditory warning is simultaneously emitted with a lateral balance between left and right cabin speakers representing the lateral position.

[0113] In some aspects, the techniques described herein relate to a method, and, when the detected object is located along a driver side of the vehicle, the camera monitor system concurrently displays the graphical warning indication in an upper left corner of a leftside display, emits an auditory warning from a left-side cabin speaker, and activates haptic feedback in a left-side region of a driver seat.

[0114] In some aspects, the techniques described herein relate to a method, and objects detected through image analysis are indicated using a first graphical icon and objects detected via vehicle-to-vehicle or vehicle-to-infrastructure communication are indicated using a second, visually distinct icon or color.

[0115] In some aspects, the techniques described herein relate to a method, the notification further including: generating at least one of the following supplemental alerts: (i) an auditory warning emitted from a vehicle sound emitter on a side corresponding to the relative location of the object; and (ii) haptic feedback to the driver through a haptic feedbackdevice, the haptic feedback modulated based on the relative location of the object; and dynamically adjusting a position or opacity of the graphical warning indication based on realtime image content to maintain compliance with field-of-view restrictions, and the location of the graphical warning indication is selected to avoid overlap with image regions depicting a trailer or road surface within a view of the camera, and detecting the object includes processing data from at least one object detection sensor selected from an ultrasonic sensor, a LIDAR sensor, or a RADAR sensor.

[0116] In some aspects, the techniques described herein relate to a method, the notification further including: generating at least one of the following supplemental alerts: (i) an auditory warning emitted from a vehicle sound emitter on a side corresponding to the relative location of the object; and (ii) haptic feedback to the driver through a haptic feedback device, the haptic feedback modulated based on the relative location of the object; and dynamically adjusting a position or opacity of the graphical warning indication based on realtime image content to maintain compliance with field-of-view restrictions, and the location of the graphical warning indication is selected to avoid overlap with image regions depicting a trailer or road surface within a view of the camera, and detecting that the warning criteria have been satisfied includes receiving a signal indicative of the object from a source external to the commercial vehicle, the signal being a vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), or vehicle-to-infrastructure (V2I) communication.

[0117] In some aspects, the techniques described herein relate to a camera monitor system (CMS) for a vehicle including a tractor and a trailer, including: a camera having an image capture unit configured to provide a field of view of the vehicle; a display configured to depict at least a portion of the field of view; and processing circuitry operatively connected to memory and configured to: detect that one or more warning criteria for notifying a driver of the vehicle about an object near the vehicle have been satisfied, the detecting including image analysis of an image recorded by a camera of the CMS; determine a relative location of the object with respect to the vehicle; and in response to the detecting, provide a notification to the driver indicating a relative location of the object with respect to the vehicle, the notification including displaying a graphical warning indication on the display at a location determined based on the relative location of the object and within a region of the display selected to comply with a legally prescribed field-of-view requirement.

[0118] In some aspects, the techniques described herein relate to a camera monitor system, and the processing circuitry is further configured to generate an auditory warning through a sound emitter located on a side of the vehicle corresponding to the relative location of the detected object.

[0119] In some aspects, the techniques described herein relate to a camera monitor system, and the auditory warning is spatialized between left and right cabin speakers to represent a lateral position of the detected object relative to a longitudinal centerline of the vehicle.

[0120] In some aspects, the techniques described herein relate to a camera monitor system, and the processing circuitry is further configured to generate haptic feedback through a haptic feedback device embedded in a driver seat or display, the haptic feedback being modulated based on the relative location or severity of the detected object.

[0121] In some aspects, the techniques described herein relate to a camera monitor system, and the processing circuitry is configured to dynamically adjust at least one of a position, brightness, or opacity of the graphical warning indication based on real-time image content of the display to maintain compliance with a legally prescribed field-of-view requirement.

[0122] In some aspects, the techniques described herein relate to a camera monitor system, and the processing circuitry is configured to display the graphical warning indication in an upper-left corner of the display corresponding to a detected object located along a driver side of the vehicle.

[0123] In some aspects, the techniques described herein relate to a camera monitor system, and the processing circuitry is configured to select the location of the graphical warning indication to avoid overlap with image regions depicting a trailer or road surface within the field of view of the camera.

[0124] In some aspects, the techniques described herein relate to a camera monitor system, and the processing circuitry is configured to display the graphical warning indication in an upper-right corner of the display corresponding to a detected object located along a passenger side of the vehicle.

[0125] In some aspects, the techniques described herein relate to a camera monitor system, and the processing circuitry is configured to determine that the warning criteria have been satisfied based on a signal received from a source external to the vehicle, the signal including at least one of a vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), or vehicle-to-infrastructure (V2I) communication.

[0126] In some aspects, the techniques described herein relate to a camera monitor system, and the processing circuitry is configured to detect the object based on data from at least one object detection sensor selected from an ultrasonic sensor, a LIDAR sensor, or a RADAR sensor.

[0127] In some aspects, the techniques described herein relate to a camera monitor system, and the processing circuitry is configured to generate the graphical warning indication including a first graphic representing the vehicle and a second graphic representing the detected object, the second graphic being positioned relative to the first graphic to indicate the relative location of the object.

[0128] Although example embodiments have been disclosed, a worker of ordinary skill in this art would recognize that certain modifications would come within the scope of the claims. For that reason, the following claims should be studied to determine their true scope and content.

Claims

CLAIMSWhat is claimed is:

1. A method for a camera monitor system (CMS) of a commercial vehicle, comprising:detecting that one or more warning criteria for notifying a driver of the commercial vehicle about an object near the vehicle have been satisfied, wherein the detecting includes image analysis of an image recorded by a camera of the CMS;determining a relative location of the object with respect to the commercial vehicle; and in response to the detecting, providing a notification to the driver indicating a relative location of the object with respect to the vehicle, the notification comprising:displaying a graphical warning indication on an electronic display of the CMS at a location determined based on the relative location of the object and within a region of the display selected to comply with a legally prescribed field-of-view requirement.

2. The method of claim 1, the notification further comprising generating an auditory warning emitted from a vehicle sound emitter on a side corresponding to the relative location of the object.

3. The method of claim 2, wherein the auditory warning is spatialized between left and right cabin speakers to represent a lateral position of the object.

4. The method of any preceding claim, the notification further comprising generating haptic feedback to the driver through a haptic feedback device, the haptic feedback modulated based on the relative location of the object.

5. The method of claim 4, wherein the haptic feedback is selectively generated in a seat region corresponding to a side of the vehicle of the object.

6. The method of any preceding claim, the notification further comprising: generating at least one of the following supplemental alerts: (i) an auditory warning emitted from a vehicle sound emitter on a side corresponding to the relative location of the object; and (ii) haptic feedback to the driver through a haptic feedback device, the haptic feedback modulated based on the relative location of the object.

7. The method of any preceding claim, comprising dynamically adjusting a position or opacity of the graphical warning indication based on real-time image content to maintain compliance with field-of-view restrictions.

8. The method of any preceding claim, wherein the graphical warning indication is displayed in an upper left corner of the electronic display corresponding to a detected object located along a driver side of the commercial vehicle.

9. The method of any preceding claim, wherein the location of the graphical warning indication is selected to avoid overlap with image regions depicting a trailer or road surface within a view of the camera.

10. The method of any preceding claim, wherein the graphical warning indication is displayed in an upper right corner of the electronic display corresponding to a detected object located along a passenger side of the commercial vehicle.

11. The method of any preceding claim, wherein detecting that the warning criteria have been satisfied comprises receiving a signal indicative of the object from a source external to the commercial vehicle, the signal being a vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), or vehicle-to-infrastructure (V2I) communication.

12. The method of any preceding claim, wherein detecting the object comprises processing data from at least one object detection sensor selected from an ultrasonic sensor, a LIDAR sensor, or a RADAR sensor.

13. The method of any preceding claim, wherein the graphical warning indication includes a first graphic representing the commercial vehicle and a second graphic representing the object, the second graphic being positioned relative to the first graphic to indicate the relative location of the object.

14. The method of any preceding claim, wherein the graphical warning indication is displayed in a corner region of the electronic display corresponding to a direction of the detected object relative to the commercial vehicle.

15. The method of any preceding claim, wherein providing the notification comprises concurrently displaying the graphical warning indication, emitting an auditory warning from a vehicle sound emitter on a side corresponding to the relative location of the object, and generating haptic feedback through a haptic feedback device, the camera monitor system dynamically adjusting at least one of a position or an opacity of the graphical warning indication based on real-time image content to maintain compliance with a legally prescribed field-of-view requirement.

16. The method of any preceding claim, wherein the graphical warning indication is displayed at a location on the electronic display corresponding to a lateral position of the detected object, and an auditory warning is simultaneously emitted with a lateral balance between left and right cabin speakers representing the lateral position.

17. The method of any preceding claim, wherein, when the detected object is located along a driver side of the vehicle, the camera monitor system concurrently displays the graphical warning indication in an upper left corner of a left-side display, emits an auditory warning from a left-side cabin speaker, and activates haptic feedback in a left-side region of a driver seat.

18. The method of any preceding claim, wherein objects detected through image analysis are indicated using a first graphical icon and objects detected via vehicle-to-vehicle or vehicle-to-infrastructure communication are indicated using a second, visually distinct icon or color.

19. The method of any preceding claim, the notification further comprising:generating at least one of the following supplemental alerts: (i) an auditory warning emitted from a vehicle sound emitter on a side corresponding to the relative location of the object; and (ii) haptic feedback to the driver through a haptic feedback device, the haptic feedback modulated based on the relative location of the object; anddynamically adjusting a position or opacity of the graphical warning indication based on real-time image content to maintain compliance with field-of-view restrictions, wherein the location of the graphical warning indication is selected to avoid overlap with image regions depicting a trailer or road surface within a view of the camera, and detecting the object comprises processing data from at least one object detection sensor selected from an ultrasonic sensor, a LIDAR sensor, or a RADAR sensor.

20. The method of any one of claims 1 to 18, the notification further comprising:generating at least one of the following supplemental alerts: (i) an auditory warning emitted from a vehicle sound emitter on a side corresponding to the relative location of the object; and (ii) haptic feedback to the driver through a haptic feedback device, the haptic feedback modulated based on the relative location of the object; anddynamically adjusting a position or opacity of the graphical warning indication based on real-time image content to maintain compliance with field-of-view restrictions, wherein the location of the graphical warning indication is selected to avoid overlap with image regions depicting a trailer or road surface within a view of the camera, and detecting that the warning criteria have been satisfied comprises receiving a signal indicative of the object from a source external to the commercial vehicle, the signal being a vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), or vehicle-to-infrastructure (V2I) communication.

Images