Telestations including associated processing capabilities and centralized teleoperations centers to support autonomous vehicle operations

Abstract

Telestations and / or a teleoperations center to support autonomous vehicle operations may comprise a processing island having one or more GPU clusters with one or more computational models to receive encoded sensor data and process the sensor data to generate vehicle commands. In addition, a centralized teleoperations center to support remote vehicle operations may comprise one or more GPU clusters that receive sensor data from vehicles, process the sensor data, generate virtual terminals based on the vehicles, and transmit the sensor data and virtual terminals to one or more virtualized telestations for presentation to a teleoperator.

Description

TELESTATIONS INCLUDING ASSOCIATED PROCESSING CAPABILITIES AND CENTRALIZED TELEOPERATIONS CENTERS TO SUPPORT AUTONOMOUS VEHICLE OPERATIONSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Application No. 63 / 742,704, filed January 7, 2025, the contents of which are herein incorporated by reference in their entirety.BACKGROUND

[0002] Teleoperated remote driving of a vehicle is considered to be an enabling technology toward fully autonomous driving. In such remote driving applications, a teleoperator may use a teleoperator station to view a live video stream representing the vehicle’s environment, and to remotely drive the vehicle via a wireless communication network. In order to facilitate safe and reliable teleoperation and / or autonomous operation of a remotely driven vehicle, various data may need to be captured and processed to safely control the vehicle, and the hardware and computational models or algorithms to perform such processing may be expensive and complex. Accordingly, there is a need for safe, reliable, and efficient processing of various data in order to support teleoperation and / or autonomous operation of such vehicles.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] FIG. l is a schematic diagram of a remote driving system including an example vehicle with a vehicle teleoperation system and an example telestation, in accordance with implementations of the present disclosure.

[0004] FIG. 2 is a schematic diagram of an example vehicle including a vehicle teleoperation system, in accordance with implementations of the present disclosure.

[0005] FIG. 3 is a schematic diagram of an example telestation with associated processing capabilities, in accordance with implementations of the present disclosure.

[0006] FIG. 4 is a schematic diagram of an example telestation and teleoperations center with associated processing capabilities, in accordance with implementations of the present disclosure.

[0007] FIG. 5 is a flow diagram illustrating an example telestation operation process, in accordance with implementations of the present disclosure.

[0008] FIG. 6 is a schematic diagram of an example centralized teleoperations center with virtualized telestations, in accordance with implementations of the present disclosure.

[0009] FIG. 7 is another schematic diagram of an example centralized teleoperations center with virtualized telestations, in accordance with implementations of the present disclosure.

[0010] FIG. 8 is a flow diagram illustrating an example virtualized telestation operation process, in accordance with implementations of the present disclosure.DETAILED DESCRIPTION

[0011] As is set forth in greater detail below, implementations of the present disclosure are directed to telestations and / or teleoperations centers that comprise hardware, software, and / or various computational models to support processing of various data for teleoperation and / or autonomous operation of remotely operated vehicles.

[0012] In example embodiments, a vehicle may comprise one or more cameras or other sensors to capture data associated with an environment proximate the vehicle, and a vehicle motion controller that communicates with various vehicle systems to instruct operation of the vehicle, e.g., throttle, braking, steering, and various peripherals. In addition, the vehicle may comprise a vehicle teleoperation system that facilitates safe and reliable communication between the remotely driven vehicle and one or more telestations and / or teleoperations center.

[0013] The vehicle teleoperation system may comprise a single, integrated unit that may be positioned or installed within the vehicle, and connected to various vehicle systems via appropriate interfaces. For example, the vehicle teleoperation system may comprise one ormore video, audio, and / or other sensor interfaces, a processing island for video and / or connectivity data, a connectivity island, a safety island, and / or one or more external interfaces to a vehicle motion controller, storage, and / or peripherals associated with the vehicle.

[0014] In some example embodiments, the vehicle teleoperation system may comprise conventional, general purpose, consumer-grade, and / or off-the-shelf computing hardware, e.g., processors, memories, and other components, that are configured to receive sensor data from cameras and sensors, encode or compress the data, and transmit the data to one or more telestations and / or a teleoperations center. In addition, the various computing hardware of the vehicle teleoperation system may receive vehicle commands from the one or more telestations and / or the teleoperations center, decode or decompress the vehicle commands, process or verify the vehicle commands, and instruct operation of the vehicle based on the vehicle commands.

[0015] In example embodiments, the one or more telestations and / or a teleoperations center may be in communication with the vehicle via a wireless communication network, and the telestations and / or teleoperations center may receive video, audio, and / or other sensor data from the vehicle. One or more processing islands of the telestations and / or a teleoperations center may comprise one or more graphics processing unit (GPU) clusters to receive and process the sensor data. Further, the GPU clusters may run or execute various computational models or algorithms upon the received sensor data, e.g., various machine learning (ML) models, artificial intelligence (Al) models or algorithms, and / or various types of deep learning, neural network, or other similar types of models, algorithms, or processing techniques. Instead of including specialized and complex hardware or software to process various sensor data onboard individual vehicles, the telestations and / or teleoperations center may comprise hardware and software to receive and perform processing of data from individual vehicles, and to generate and transmit outputs for remote operation of the vehicles, e.g., autonomous vehicle operation.

[0016] The outputs from the processing islands and GPU clusters, e.g., processed sensor data, may also be presented or provided to a teleoperator at a telestation, in order to enable remote teleoperation of the vehicle. Based on the presented data, the telestation may receive inputs or commands from the teleoperator, which inputs or commands may then be communicated to the vehicle via the wireless communication network in order to remotely operate or drive the vehicle.

[0017] In additional example embodiments, a centralized teleoperations center may be in communication with the vehicle via a wireless communication network, and the centralized teleoperations center may receive video, audio, and / or other sensor data from the vehicle. One or more processing islands of the centralized teleoperations center may comprise one or more graphics processing unit (GPU) clusters to receive and process the sensor data. Further, the GPU clusters may run or execute various computational models or algorithms upon the received sensor data, e.g., various machine learning (ML) models, artificial intelligence (Al) models or algorithms, and / or various types of deep learning, neural network, or other similar types of models, algorithms, or processing techniques.

[0018] The outputs from the processing islands and GPU clusters of the centralized teleoperations center, e.g., processed sensor data, may then be transmitted or provided to one or more virtualized telestations and associated teleoperators, in order to enable remote teleoperation of the vehicle. The virtualized telestations may comprise conventional, general purpose, consumer-grade, and / or off-the-shelf computing hardware, e.g., processors, memories, and other components, that are configured to receive processed sensor data from the centralized teleoperations center, decode or decompress the data, and present or provide the data to the teleoperator. Instead of including specialized and complex hardware or software associated with particular vehicle platforms, classes, or types, the virtualized telestations may be provisioned or reconfigured for various different vehicle platforms, classes, or types as needed, e.g., based on data and software received or requested from the centralized teleoperations center.

[0019] Based on the presented data, the virtualized telestation may receive inputs or commands from the teleoperator, which inputs or commands may be transmitted to and processed by the centralized teleoperations center, e.g., by an associated safety island. Then, the inputs or commands may be communicated to the vehicle via the wireless communication network in order to remotely operate or drive the vehicle.

[0020] Using the example vehicles including onboard vehicle teleoperation systems and one or more telestations and / or a teleoperations center having associated processing capabilities described herein that are in communication with each other, cost and complexity of hardware and / or software that is onboard remotely operated vehicles may be reduced. Further, using a centralized teleoperations center and one or more virtualized telestations described herein, additional cost and complexity of hardware and / or software associated withdedicated or specialized telestations may be further reduced. Moreover, safe and reliable teleoperation and / or autonomous operation of vehicles may be enabled for various platforms, classes, types, or configurations of vehicles.

[0021] FIG. 1 is a schematic diagram 100 of a remote driving system including an example vehicle with a vehicle teleoperation system and an example telestation, in accordance with implementations of the present disclosure.

[0022] As shown in FIG. 1, the example remote driving system may comprise a vehicle 102 that is adapted to be remotely driven, controlled, or instructed by a teleoperator via a wireless communication network 105, e.g., the Internet, and / or that may be adapted for at least partially autonomous operations. In addition, the example remote driving system may comprise a teleoperator station or telestation 110 for use by a teleoperator to remotely drive, control, or instruct the vehicle 102 via the wireless communication network 105.

[0023] In example embodiments, the vehicle 102 may comprise a car, such as a small car, a regular car, a Sports Utility Vehicle (SUV), a van, a truck, or any other type of commercial, industrial, or personal vehicle that is adapted to be remotely driven, controlled, or instructed. The vehicle 102 may comprise or include various on-board infrastructure to enable or facilitate teleoperation, e.g., cameras 103, various imaging sensors, audio sensors, proximity or ranging sensors, other types of sensors, sensor processing units, and / or vehicle motion controller 104. For example, the vehicle 102 may include one or more imaging devices, cameras, or other sensors 103 for capturing imaging data of the vehicle's environment, and / or one or more audio sensors or arrays, radar (radio detection and ranging) sensors, LIDAR (light detection and ranging) sensors, or other types of sensors for detecting or capturing data associated with the vehicle’s environment. In addition, the vehicle motion controller 104 may interface or connect with various vehicle systems, e.g., throttle, braking, steering, and / or various peripherals or accessories, to remotely operate the vehicle 102.

[0024] The imaging devices or cameras 103 associated with the vehicle 102 may comprise various types of imaging sensors, analog cameras, digital cameras, video cameras, depth sensors, infrared sensors, time-of-flight sensors, or other types of imaging sensors. The imaging devices or cameras 103 may be positioned and oriented at various positions on the vehicle 102 in order to capture imaging data of an environment at least partially around the vehicle 102, e.g., towards a forward movement direction, towards a rearward movementdirection, and / or toward various other portions of a periphery of the vehicle 102. In addition, the imaging devices or cameras 103 may capture imaging data, such as video data, live video streams, or other types of imaging data, which may be processed and transmitted to the teleoperator station 110 and used to facilitate remote operation of the vehicle 102, as further described herein.

[0025] The audio sensors or arrays associated with the vehicle 102 may comprise various types of microphones, microphone arrays, audio transducers, piezoelectric elements, and / or other types of audio sensors. The audio sensors or arrays may be positioned and oriented at various positions on the vehicle 102 in order to detect and capture audio data of an environment at least partially around the vehicle 102. In some examples, an audio sensor array or microphone array may be beamformed to detect and capture audio data at particular desired positions or locations relative to the vehicle 102. In addition, the audio sensors or arrays may capture audio data, such as voices, speech, footsteps, bicycles, tire or road noise, vehicles, engines, motors, or other types of sounds or audio data, which may be processed and transmitted to the teleoperator station 110 to facilitate remote operation of the vehicle 102, as further described herein.

[0026] The vehicle motion controller 104 may interface or connect with various vehicle systems, e.g., throttle, braking, steering, and / or various peripherals or accessories, to remotely operate the vehicle 102. For example, the vehicle motion controller 104 may comprise one or more sensors to detect or measure drive state information, and / or may receive data from such sensors. In addition, the vehicle motion controller 104 may couple or connect with various vehicle systems, e.g., physically, mechanically, electrically, or otherwise, and provide instructions or commands to enable or cause remote operation of the vehicle 102.

[0027] The sensors to detect or measure drive state information of the vehicle 102 may comprise various types of sensors configured to detect speed, acceleration, steering angle, yaw rate, steering torque, and / or other operational characteristics of the vehicle 102. For example, a first sensor such as a speedometer or encoder may measure a drive speed of the vehicle 102, a second sensor such as an accelerometer, pressure sensor, or encoder may measure pedal actuation, acceleration, deceleration, or braking of the vehicle 102, and / or a third sensor such as an encoder or position / orientation sensor may measure a steering angle, yaw rate, steering torque, and / or measure an orientation of the vehicle wheels. The drivestate information of the vehicle 102 may be processed and used by the vehicle motion controller 104 to facilitate remote operation of the vehicle 102, as further described herein.

[0028] The vehicle motion controller 104 may transmit instructions or commands to various vehicle systems to remotely operate the vehicle 102. For example, the vehicle motion controller 104 may communicate or couple directly with various actuators, subsystems, or systems of the vehicle 102, such as mechanical actuators that directly actuate the vehicle's steering wheel, acceleration pedal, brakes, and / or other systems, components, or peripherals of the vehicle 102. Alternatively, the vehicle motion controller 104 may communicate with existing actuators of the vehicle 102 via one or more electrical interfaces (e.g., for adjusting or controlling speed, acceleration, steering angle, peripheral or accessory functions, and / or other operational characteristics) to control functions or operations of the vehicle 102.

[0029] As shown in FIG. 1, an example vehicle teleoperation system 115 may be installed or assembled within the vehicle 102. The vehicle teleoperation system 115 may comprise a single, integrated unit that includes one or more camera interfaces 116, other sensor interfaces, one or more processing islands 117 for video data and / or connectivity data, a connectivity island 118, a safety island 119, and one or more external interfaces 120 that communicate or couple with the vehicle motion controller 104, storage, peripherals, and / or other components or subsystems of the vehicle 102.

[0030] In example embodiments, the vehicle teleoperation system 115 may receive, via the camera interfaces 116 and other sensor interfaces, data from various cameras 103 and sensors associated with the vehicle 102. A processing island 117 for video and other sensor data may process the imaging data and other sensor data for transmission to and further processing and presentation by a teleoperator station 110. The connectivity island 118 may transmit and receive data to and from the teleoperator station 110 via one or more networks 105, e.g., transmit video and other sensor data to the teleoperator station 110, and receive user input, commands, or instructions from the teleoperator station 110.

[0031] Then, a processing island 117 for connectivity data may receive and process the user input, commands, or instructions, and may forward the user input to the safety island 119. The safety island 119 may further process and / or verify the user input, e.g., for accuracy, reliability, integrity, latency, or other aspects, to determine whether to instruct the vehicle 102 based on the received user input. Further, the vehicle teleoperation system 115may transmit, via the external interfaces 120, user input, commands, instructions, or other data to the vehicle motion controller 104 to remotely operate the vehicle 102, as well as transmit data to storage, peripherals, accessories, or other subsystems of the vehicle 102.

[0032] Further details of the portions, components, or subsystems of the vehicle teleoperation system 115 and additional example embodiments are described in U.S.Application No. 18 / 765,733, filed by Applicant on July 8, 2024, the contents of which are herein incorporated by reference in their entirety.

[0033] As further shown in FIG. 1, the wireless communication network 105 may comprise a network that allows for bi-directional transmission of data between the vehicle 102 and the teleoperator station 110. For example, the network 105 may be the Internet, a fourth generation (4G) wireless communication network, a fifth generation (5G) wireless communication network, various cellular or satellite communication networks, or other types of wireless communication networks.

[0034] Various data or information may be transmitted via the network 105, including imaging data, audio data, other sensor data, location data, vehicle data, and / or various other data associated with the vehicle 102, e.g., from the vehicle 102 to the teleoperator station 110, as well as drive control inputs, commands, or instructions, and / or other data, information, commands, or instructions, e.g., from the teleoperator station 110 to the vehicle 102 via the wireless communication network 105. For example, imaging and / or sensor data may be transmitted from the vehicle 102 via the network 105 to the teleoperator station 110 for processing and presentation by a display, monitor, screen, or other presentation device, e.g., the camera view 112. In addition, various user inputs, commands, or instructions 114 to remotely operate the vehicle 102 may be transmitted from the teleoperator station 110 via the network 105 to the vehicle 102. Further, various additional data may be exchanged between the vehicle 102 and the teleoperator station 110, such as time synchronization or latency information, data transmission timestamps, sequence indicators or identifiers, formatting information, and / or various other data or metadata.

[0035] In example embodiments, the teleoperator station 110 may comprise a communication unit configured to send and receive data or information to and from the vehicle 102 via the network 105, one or more processors or processing units configured to process various data such as imaging data, sensor data, user inputs, or others, a presentationor display device, e.g., camera view 112, configured to present, emit, or provide the imaging data, audio data, or other sensor data associated with a vehicle and its environment, and various input devices, e.g., keyboards, mice, touchscreens, touchpads, steering or control wheels, pedals, buttons, knobs, or other user interface elements, configured to receive user inputs, commands, or instructions 114 from the teleoperator using the teleoperator station 110.

[0036] The communication unit may comprise various types of communication systems, devices, antenna, interfaces, or other data transmit / receive units configured to enable wireless communication between the teleoperator station 110 and the vehicle 102 via the wireless communication network 105. As described herein, the communication unit may receive imaging data, audio data, other sensor data, location data, vehicle data, and / or various other data from the vehicle 102, and may transmit drive control inputs, commands, or instructions, and / or other data to the vehicle 102.

[0037] The processors may comprise one or more processing units, graphics processing units (GPUs), GPU clusters, or other types of processors configured to process the various data that is received and / or sent between the vehicle 102 and teleoperator station 110 via the network 105. For example, the processors may receive and process various imaging data, audio data, or other sensor data from the vehicle 102, and the processors may process and transmit various user inputs, commands, or instructions to the vehicle 102.

[0038] In further example embodiments, the teleoperator station 110 may also comprise various components, processors, islands, or other portions similar to the vehicle teleoperation system 115 that is installed within the vehicle 102. For example, the teleoperator station 110 may comprise one or more processing islands for video data and / or connectivity data, a connectivity island, a safety island, and / or one or more external interfaces that communicate or couple with the vehicle 102.

[0039] For example, a processing island of the teleoperator station 110 for video and other sensor data may process imaging data and other sensor data that is received from a vehicle teleoperation system 115 of a vehicle 102. A connectivity island of the teleoperator station 110 may receive and transmit data to and from the vehicle teleoperation system 115 of a vehicle 102 via one or more networks 105, e.g., receive video and other sensor data at the teleoperator station 110 from the vehicle teleoperation system 115 of a vehicle 102, andtransmit user input, commands, or instructions to the vehicle teleoperation system 115 of a vehicle 102.

[0040] Then, a processing island of the teleoperator station 110 for connectivity or user input data may receive and process the user input, commands, or instructions, and may forward the user input to a safety island of the teleoperator station 110. The safety island may further process and / or verify the user input, e.g., for accuracy, reliability, integrity, latency, or other aspects, to determine whether to transmit instructions to the vehicle 102 based on the received user input. Further, the connectivity island of the teleoperator station 110 may transmit, via the external interfaces, user input, commands, instructions, or other data to the vehicle teleoperation system 115 of a vehicle 102 to remotely operate the vehicle 102, e.g., via the vehicle motion controller 104, as well as transmit data to storage, peripherals, accessories, or other subsystems of the vehicle 102.

[0041] The processing islands, connectivity islands, safety islands, and / or external interfaces of the teleoperator station 110 may perform any and all of the functions described herein with respect to processing islands 117, connectivity islands 118, safety islands 119, and / or external interfaces 120 of various example embodiments of the vehicle teleoperation systems 115 presented and described herein. In addition, various of the functions or operations may be split, divided, shared, or otherwise distributed among the various islands of the example teleoperator stations 110 and example vehicle teleoperation systems 115.

[0042] In example embodiments described herein, the processing islands 117 of the telestations 110 may comprise various hardware, software, and / or computational models to process the various sensor data received from a vehicle 102, whereas the processing islands 117 of the vehicle 102 may comprise consumer-grade, off-the-shelf hardware to perform basic or limited processing of the sensor data, e.g., encoding or compressing the sensor data for transmission. In some examples, the processing islands 117 of the telestations 110 may comprise one or more GPU clusters that can run or execute various ML models, Al models or algorithms, and / or other computational models to process the sensor data received from the vehicle 102. In other examples, the processing islands 117 may be associated with a centralized teleoperations center that comprises one or more GPU clusters that can perform various data processing, and also run or execute various ML models, Al models or algorithms, and / or other computational models to process the sensor data received from the vehicle 102, and the centralized teleoperations center may be in communication with one ormore virtualized telestations to present the processed sensor data to a teleoperator and receive vehicle commands from the teleoperator. Further details of the portions, components, or subsystems of various processing islands, connectivity islands, safety islands, and / or external interfaces of the teleoperator station 110 are described herein at least with respect to FIGs. 3-8.

[0043] The camera view 112 may comprise one or more monitors, screens, projectors, display devices, head-mounted displays, augmented reality displays, other types of presentation devices, speakers, audio output devices, haptic feedback or output devices, and / or other types of feedback or output devices. For example, the camera view 112 may receive and present, render, or display the imaging data, e.g., video data or live video streams, received from the vehicle 102. In addition, the camera view 112 may receive and emit sounds or other audio data associated with objects in a vehicle’s environment. Moreover, the camera view 112 may emit various other information, indicators, or feedback, e.g., visual, audio, haptic, or other types of feedback, based on the received sensor data. The camera view 112 may present, emit, or provide the various imaging data, audio data, information, feedback, or indicators, such that a teleoperator at the teleoperator station 110 may have an awareness of the vehicle 102 and an environment around the vehicle 102 in order to remotely operate the vehicle 102.

[0044] The input devices may comprise a steering wheel, acceleration pedal, brake pedal, transmission selector, and / or various other interface components to generate drive control inputs or commands 114 for the vehicle 102. In addition, the input devices may include components, elements, or interfaces to control or instruct various other aspects of the vehicle 102, such as lights, turn indicators, windshield wipers, power windows, power doors, climate control systems, entertainment or infotainment systems, and / or various other systems, devices, or accessories associated with the vehicle 102. The input devices may receive drive control inputs, commands, or instructions 114 provided or input by a teleoperator at the teleoperator station 110, which may then be processed and / or transmitted to the vehicle 102 via the network 105 in order to remotely operate the vehicle 102.

[0045] Although FIG. 1 illustrates an example remote driving system having a particular number, type, configuration, and arrangement of various components, other example embodiments may include various other numbers, types, configurations, and arrangements of the various components. For example, vehicles may have various numbers, types,configurations, or arrangement of cameras, sensors, or vehicle motion controllers, vehicles may have various example embodiments of vehicle teleoperation systems described herein, one or more vehicles may be in communication with one or more teleoperator stations, various types of wireless communication networks may be used to facilitate communication between vehicles and teleoperator stations, telestations and / or teleoperations centers may have various example embodiments of processing islands, safety islands, or other components or subsystems described herein, and / or various other modifications may be made in other example embodiments of the example remote driving system.

[0046] FIG. 2 is a schematic diagram 200 of an example vehicle including a vehicle teleoperation system, in accordance with implementations of the present disclosure.

[0047] The example vehicle 102 illustrated in FIG. 2 may include any and all of the features of the vehicle 102 described herein at least with respect to FIG 1.

[0048] For example, the vehicle 102 may include various types of sensors to detect or capture data associated with various objects or portions of an environment around the vehicle, including one or more imaging devices, cameras, or sensors 103 for capturing imaging data of the vehicle's environment, and / or one or more audio sensors or arrays, radar sensors, LIDAR sensors, or other types of sensors for detecting or capturing data associated with the vehicle’s environment.

[0049] In example embodiments, the imaging devices or cameras 103 associated with the vehicle 102 may comprise various types of imaging sensors, analog cameras, digital cameras, video cameras, depth sensors, infrared sensors, time-of-flight sensors, or other types of imaging sensors. The imaging devices or cameras 103 may be positioned and oriented at various positions on the vehicle 102 in order to capture imaging data of an environment at least partially around the vehicle 102, e.g., towards a forward movement direction, towards a rearward movement direction, and / or toward various other portions of a periphery of the vehicle 102. In addition, the imaging devices or cameras may capture imaging data, such as video data, live video streams, or other types of imaging data, which may be transmitted to the teleoperator station 110 and used to facilitate remote operation of the vehicle 102, as further described herein.

[0050] In the example of FIG. 2, the imaging devices 103 may be positioned toward a forward portion of the vehicle 102 in order to capture imaging data of an environment towarda forward movement direction of the vehicle 102, and some imaging devices 103 may be positioned toward a side portion of the vehicle 102 in order to capture imaging data of an environment toward one or more sides of the vehicle 102. Although not illustrated in FIG. 2, various additional imaging devices may be positioned at other portions of the vehicle 102 to capture imaging data of the environment toward other directions relative to the vehicle, e.g., toward a rearward movement direction, toward lateral sides or corners of the vehicle, or any other directions.

[0051] Furthermore, imaging data that is captured by the imaging devices 103 may be processed to identify objects, and also to identify locations of objects within the imaging data relative to the vehicle 102. For example, the relative locations of objects within imaging data may be determined based on known positions, orientations, and fields of view of the imaging devices 103 relative to the vehicle.

[0052] In example embodiments, audio sensors or arrays associated with the vehicle 102 may comprise various types of microphones, microphone arrays, audio transducers, piezoelectric elements, and / or other types of audio sensors. The audio sensors or arrays may be positioned and oriented at various positions on the vehicle 102, similar to the positions and orientations of the imaging devices 103, in order to detect and capture audio data of an environment at least partially around the vehicle 102, e.g., towards a forward movement direction, towards a rearward movement direction, and / or toward various other portions of a periphery of the vehicle 102. In addition, the audio sensors or arrays may capture audio data, such as voices, speech, footsteps, bicycles, tire or road noise, vehicles, engines, motors, or other types of sounds or audio data, which may be transmitted to the teleoperator station 110 and used to facilitate remote operation of the vehicle 102, as further described herein.

[0053] Furthermore, audio data that is captured by an audio sensor array or microphone array may be processed to identify sounds, and also to identify locations of objects associated with the sounds in the environment relative to the vehicle 102. For example, the relative locations of objects in the environment may be determined based on known positions and orientations of individual audio sensors or microphones of an array relative to the vehicle, as well as relative times of receipt of audio data by the individual audio sensors or microphones of an array.

[0054] In example embodiments, various other sensors associated with the vehicle 102 may comprise various types of depth sensors, radar sensors, LIDAR sensors, or other types of time-of-flight sensors. The sensors may also be positioned and oriented at various positions on the vehicle 102, similar to the positions and orientations of the imaging devices 103, in order to capture data of an environment at least partially around the vehicle 102, e.g., towards a forward movement direction, towards a rearward movement direction, and / or toward various other portions of a periphery of the vehicle 102. In addition, the sensors may capture various types of data, which may be transmitted to the teleoperator station 110 and used to facilitate remote operation of the vehicle 102, as further described herein.

[0055] Furthermore, data that is captured by the other sensors may be processed to identify distances or ranges to objects, and also to identify locations of objects within the data relative to the vehicle 102. For example, the relative locations of objects within the data may be determined based on known positions, orientations, and fields of sensing or view of the sensors relative to the vehicle.

[0056] In addition, as shown in FIG. 2, the vehicle 102 may include a vehicle motion controller 104 that is configured to interface with various vehicle systems to control functions and operations of the vehicle 102, including throttle, braking, steering, and various peripherals or accessories. Generally, vehicle motion controllers 104 may be specific or specialized for particular vehicle types or vehicle platforms. For example, a first vehicle or component manufacturer may design and manufacture a first vehicle motion controller that is specialized for particular first vehicles or platforms, whereas a second vehicle or component manufacturer may design and manufacture a second vehicle motion controller that is specialized for particular second vehicles or platforms. As a result, different vehicle motion controllers may require or receive different types, attributes, formatting, software, specifications, or other aspects related to user inputs, commands, or instructions that may be generated and received from teleoperator stations to remotely operate respective vehicles.

[0057] In example embodiments, the vehicle 102 may also comprise an example vehicle teleoperation system 115 that is installed or assembled in the vehicle 102. The vehicle teleoperation system 115 may communicate or interface with the various imaging sensors or devices 103, audio sensors, or other types of sensors onboard the vehicle 102. For example, the vehicle teleoperation system 115 may receive various sensor data, process the sensor data,and then transmit the processed sensor data to a teleoperator station via one or more networks 105.

[0058] In addition, the vehicle teleoperation system 115 may communicate or interface with the vehicle motion controller 104 that is present onboard the vehicle 102. In order to effectively communicate with the vehicle motion controller 104, the vehicle teleoperation system 115 may be provisioned, configured, adapted, or modified based on the vehicle, vehicle type, platform, class, make, model, software, or other attributes, characteristics, or aspects of the vehicle 102. For example, the vehicle teleoperation system 115 may receive user inputs, commands, or instructions from a teleoperator station via one or more networks 105, process, provision, adapt, and / or verify the user inputs, and then forward the user inputs to the vehicle motion controller 104 to instruct remote operation of the vehicle 102.

[0059] Although FIG. 2 illustrates an example vehicle having a particular number, type, configuration, and arrangement of various components, other example embodiments may include various other numbers, types, configurations, and arrangements of the various components. For example, vehicles may have various numbers, types, configurations, or arrangement of cameras, sensors, or vehicle motion controllers, vehicles may have various example embodiments of vehicle teleoperation systems described herein, various types of wireless communication networks may be used to facilitate communication between vehicles and teleoperator stations, and / or various other modifications may be made in other embodiments of the example vehicle.

[0060] FIG. 3 is a schematic diagram 300 of an example telestation with associated processing capabilities, in accordance with implementations of the present disclosure.

[0061] Conventional autonomous vehicles and / or conventional autonomous vehicle architectures are generally equipped with high performance compute units, related hardware, software, and / or computational models to process various sensor data, and to generate vehicle commands onboard individual vehicles. Such high performance compute units, related hardware, software, and / or computational models are generally very expensive, difficult to maintain or update, and customized for particular vehicle types, platforms, or classes. For example, the high performance compute units onboard individual vehicles may comprise one or more GPUs or GPU clusters that run or execute various types of computational models or algorithms.

[0062] In addition, the computational models onboard individual vehicles may comprise various ML or Al models that are configured and customized to process various sensor data from cameras, microphones, ranging sensors, and others. Moreover, because of size, capacity, and other constraints associated with hardware equipped onboard individual vehicles, the computational models may be modified, optimized, or tailored based on such constraints. However, as a result of such modification, the computational models may suffer from bias due to model simplification, tradeoffs between speed and accuracy, batch processing limitations, reduced feature representations, and / or other drawbacks or limitations.

[0063] In example embodiments described herein, the high performance compute units, related hardware, software, and / or computational models may be associated with one or more telestations and / or a teleoperations center, instead of including such components, features, and models onboard individual vehicles. Combined with one or more safety islands to ensure safe operation of remotely driven vehicles responsive to network connectivity issues, the telestations and / or teleoperations center described herein may resolve or minimize many of the drawbacks and limitations associated with conventional autonomous vehicle architectures.

[0064] For example, using example embodiments of the telestations and / or teleoperations center as described herein, modifications to existing vehicles to enable remote driving operations can be simplified and made more cost effective, utilization of high performance compute units, related hardware, software, and / or computational models may be increased or improved, various computational models may not need to be modified or optimized based on vehicle constraints, upgrades and updates to high performance compute units, related hardware, software, and / or computational models may be more easily incorporated, and / or various other benefits or advantages may be realized.

[0065] By incorporating high performance compute units, related hardware, software, and / or computational models into telestations and / or teleoperations center instead of onboard individual vehicles, various hardware and software to be installed in individual vehicles may be simplified and minimized, thereby reducing time and cost associated with enabling remote operation of vehicles. For example, the vehicle teleoperation system 115 to be installed within individual vehicles may perform basic or limited processing of various sensor data. In some examples, the processing island 117 of a vehicle teleoperation system 115 may comprise consumer-grade and / or off-the-shelf hardware that receives and processes thesensor data, e.g., encodes or compresses the sensor data for transfer to the telestations and / or teleoperations center.

[0066] Furthermore, hardware and software installed onboard individual vehicles may experience vibrations, shocks, temperature and humidity changes, and / or various other environmental or external factors. In order to withstand such operational vibrations, changes, and factors, any hardware and software installed onboard individual vehicles may be ruggedized and further tested to ensure operability in such conditions. By incorporating high performance compute units, related hardware, software, and / or computational models into telestations and / or teleoperations center, e.g., associated with controlled environments, instead of onboard individual vehicles, the time, effort, and cost associated with ruggedizing hardware and software to be installed onboard individual vehicles may be minimized or eliminated.

[0067] In addition, by incorporating high performance compute units, related hardware, software, and / or computational models into telestations and / or teleoperations center instead of onboard individual vehicles, utilization of such hardware and software may be increased or improved because multiple vehicles may communicate and interact with the telestations and / or teleoperations center, such that the hardware and software of the telestations and / or teleoperations center are effectively shared among multiple vehicles over time. In contrast, conventional autonomous vehicle architectures having high performance compute units, hardware, and software equipped onboard individual vehicles only utilize such hardware and software when the particular individual vehicles are in operation. Moreover, as utilization of hardware and software of telestations and / or teleoperations center increases, additional capacity, e.g., additional high performance compute units and related hardware, may be gradually scaled up or increased to support remote operation of a greater number of vehicles, telestations, and / or teleoperations centers.

[0068] Further, because the telestations and / or teleoperations center do not have the same size, capacity, and other constraints as individual vehicles, software and computational models that are run or executed to process various sensor data do not need to be modified, optimized, or tailored based on such constraints, e.g., full feature set models or algorithms may be utilized. This may reduce or prevent the creation of various drawbacks or limitations, including bias due to model simplification, tradeoffs between speed and accuracy, batch processing limitations, reduced feature representations, and / or others. Moreover, upgradesand / or updates to various software and computational models may be incorporated more quickly and uniformly for multiple telestations and / or a teleoperations center, as opposed to multiple different types of vehicles with different optimized versions of software and computational models. In addition, additional models, inferences, feature sets, or other computational attributes or aspects may be tested and / or added over time to the software and computational models of the telestations and / or teleoperations center, without having to upgrade or update software or computational models of individual vehicles.

[0069] As shown in FIG. 3, an example telestation 110 with associated processing capabilities may comprise a connectivity island 318, a safety island 319, external interfaces 320, and a processing island 322. The connectivity island 318, the safety island 319, and external interfaces 320 may include any and all of the features of similar components of the telestation 110 described herein at least with respect to FIG. 1. For example, the connectivity island 318 may receive various sensor data from a vehicle, and may transmit vehicle commands to a vehicle, e.g., via network 105. In addition, the safety island 319 may receive vehicle commands from the processing island 322, and may process or verify such vehicle commands prior to transmission to a vehicle.

[0070] The various sensor data that is received by the connectivity island 318 may be at least partially processed by a vehicle teleoperation system of the vehicle. For example, the basic or limited processing onboard the vehicle may comprise encoding or compressing the various sensor data, in order to transmit the sensor data to the telestation 110 via the network 105.

[0071] In example embodiments, the processing island 322 may form a portion of or otherwise be associated with the telestation 110, and the processing island 322 may comprise one or more GPU clusters 324. In addition, each GPU cluster may be configured to run multiple computational models or inferences for multiple vehicle types, classes, or platforms. For example, a single GPU cluster may be configured for or capable of receiving and processing sensor data for five, ten, or other numbers of vehicles at a same time.

[0072] In addition, individual GPU clusters 324 may comprise various computational models 326, e.g., ML models, Al models or algorithms, deep learning models, neural networks, or various other types of models, algorithms, or processing techniques. The computational models 326 may perform various processing of the sensor data, includingimage segmentation, feature or object detection, object, obstacle, or path identification, potential path generation, desired path selection, vehicle commands generation, and / or other sensor data processing.

[0073] For example, upon receiving sensor data from a vehicle, the processing island 322 and / or GPU clusters 324 may decode or decompress the received sensor data. In some examples, the sensor data may comprise imaging data, audio data, time-of-flight data, and / or other sensor data. Then, the processing island 322 and / or GPU clusters 324 may process the sensor data using one or more computational models 326. The computational models 326 may generate various outputs, including object or obstacle identifications and / or locations, one or more potential paths, a desired or selected path, one or more vehicle commands associated with respective paths, and / or other outputs to enable teleoperation and / or autonomous operation of the vehicle.

[0074] The processing island 322 and / or GPU clusters 324 may further encode or compress the outputs from the computational models 326, and then transmit, via the connectivity island 318, the outputs to the vehicle via the network 105. In some examples, the safety island 319 may further receive and process the outputs from the computational models 326 in order to verify various aspects or attributes, e.g., accuracy, reliability, integrity, latency, or other aspects, of the objects, paths, vehicle commands, or other outputs. Upon verifying the outputs, the processing island 322 and / or the safety island 319 may transmit, via the connectivity island 318, the outputs to the vehicle via the network 105.

[0075] FIG. 4 is a schematic diagram 400 of an example telestation and teleoperations center with associated processing capabilities, in accordance with implementations of the present disclosure.

[0076] As shown in FIG. 4, one or more example telestations 110 with associated processing capabilities may each comprise a connectivity island 318, a safety island 319, and external interfaces 320. The connectivity island 318, the safety island 319, and external interfaces 320 may include any and all of the features of similar components of the telestation 110 described herein at least with respect to FIG. 1. For example, the connectivity island 318 may receive various sensor data from a vehicle, and may transmit vehicle commands to a vehicle, e.g., via network 105. In addition, the safety island 319 may receive vehiclecommands from the processing island 432 of a teleoperations center 430, and may process or verify such vehicle commands prior to transmission to a vehicle.

[0077] The various sensor data that is received by the connectivity island 318 may be at least partially processed by a vehicle teleoperation system of the vehicle. For example, the basic or limited processing onboard the vehicle may comprise encoding or compressing the various sensor data, in order to transmit the sensor data to the telestation 110 via the network 105.

[0078] In example embodiments, a teleoperations center 430 may be located physically near or at a location with one or more telestations 110. Alternatively, the teleoperations center 430 may be separate or remote from one or more telestations 110, e.g., similar to a cloud computing server or data center. In some examples, the teleoperations center 430 may comprise one or more data centers that are positioned nearby or within regions in which multiple telestations 110 may operate and communicate with the teleoperations center 430.

[0079] In addition, the teleoperations center 430 may communicate with the one or more telestations 110 via various connection or communication methods, standards, or technologies. For example, the communication methods may comprise wired or fiber connections, ethemet connections, the Internet, wireless communication protocols, or other types of communication methods. In some examples, wired, fiber, ethernet, or similar connections may be preferred, in order to reduce or minimize latency associated with transfer of data between the teleoperations center 430 and one or more telestations 110.

[0080] In example embodiments, the teleoperations center 430 may comprise a processing island 432 having one or more GPU clusters 434. In addition, each GPU cluster may be configured to run multiple computational models or inferences for multiple vehicle types, classes, or platforms. For example, a single GPU cluster may be configured for or capable of receiving and processing sensor data for five, ten, or other numbers of vehicles at a same time. Moreover, the one or more GPU clusters 434 may be configured for or capable of receiving and processing sensor data for five, ten, or other numbers of telestations 110 at a same time, thereby enabling further sharing of hardware and software of the teleoperations center 430 among multiple telestations 110, as well as multiple vehicles.

[0081] In addition, individual GPU clusters 434 may comprise various computational models 436, e.g., ML models, Al models or algorithms, deep learning models, neuralnetworks, or various other types of models, algorithms, or processing techniques. The computational models 436 may perform various processing of the sensor data, including image segmentation, feature or object detection, object, obstacle, or path identification, potential path generation, desired path selection, vehicle commands generation, and / or other sensor data processing.

[0082] For example, upon receiving sensor data from a vehicle via a telestation 110, the processing island 432 and / or GPU clusters 434 of the teleoperations center 430 may decode or decompress the received sensor data. In some examples, the sensor data may comprise imaging data, audio data, time-of-flight data, and / or other sensor data. Then, the processing island 432 and / or GPU clusters 434 may process the sensor data using one or more computational models 436. The computational models 436 may generate various outputs, including object or obstacle identifications and / or locations, one or more potential paths, a desired or selected path, one or more vehicle commands associated with respective paths, and / or other outputs to enable teleoperation and / or autonomous operation of the vehicle.

[0083] The processing island 432 and / or GPU clusters 434 may further encode or compress the outputs from the computational models 436, and transmit the outputs to the telestation 110, which may then transmit, via the connectivity island 318, the outputs to the vehicle via the network 105. In some examples, the safety island 319 may further receive and process the outputs from the computational models 436 in order to verify various aspects or attributes, e.g., accuracy, reliability, integrity, latency, or other aspects, of the objects, paths, vehicle commands, or other outputs. Upon verifying the outputs, the safety island 319 may transmit, via the connectivity island 318, the outputs to the vehicle via the network 105.

[0084] FIG. 5 is a flow diagram illustrating an example telestation operation process 500, in accordance with implementations of the present disclosure.

[0085] The process 500 may begin by receiving imaging data captured by vehicle cameras using a telestation connectivity island, as at 502. For example, one or more cameras or other sensors onboard a vehicle may capture sensor data, and a processing island of the vehicle may process, e.g., encode or compress, the sensor data for transmission to a telestation using a connectivity island of the vehicle. Then, a telestation connectivity island may receive the imaging and other sensor data from the vehicle. Further, a control system may instruct receiving the imaging and sensor data by the telestation connectivity island.

[0086] The process 500 may continue by decompressing the imaging data using a telestation processing island, as at 504. For example, a processing island, e.g., associated with the telestation or with a teleoperations center in communication with the telestation, may receive the imaging and other sensor data from the connectivity island. Then, the telestation processing island may process, e.g., decode or decompress, the sensor data for further processing. Further, a control system may instruct decompressing the imaging and sensor data by the telestation processing island.

[0087] The process 500 may proceed by processing the imaging data using ML / Al models of GPU clusters, as at 506. For example, the telestation processing island may comprise one or more GPU clusters having various computational models, e.g., ML models, Al models, or other models, algorithms, or processing techniques. The GPU clusters may receive and process the imaging and sensor data using one or more computational models, which may generate various outputs such as object or feature identifications, potential paths, selected paths, vehicle commands, or other outputs. Further, a control system may instruct processing the imaging and sensor data using various computational models of the telestation processing island.

[0088] The process 500 may continue to generate vehicle commands using the GPU clusters, as at 508. For example, based on processing of the imaging and sensor data using one or more computational models, various outputs may be generated by the GPU clusters, including object or feature identifications, potential paths, selected paths, vehicle commands, or other outputs. Further, a control system may instruct generating vehicle commands using the various computational models of the telestation processing island.

[0089] The process 500 may proceed to process the vehicle commands using a telestation safety island, as at 510. For example, the vehicle commands or other outputs may be received by the telestation safety island, and processed to verify various aspects of the outputs, e.g., for accuracy, reliability, integrity, latency, or other aspects. The safety island may ensure that any commands or instructions provided to the vehicle are checked or verified to ensure safe operation of the vehicle, whether teleoperation or autonomous operation. Further, a control system may instruct processing the vehicle commands by a telestation safety island.

[0090] The process 500 may continue with compressing the vehicle commands using the telestation processing island, as at 512. For example, the processing island, e.g., associated with the telestation or with a teleoperations center in communication with the telestation, may process, e.g., encode or compress, the vehicle commands or other outputs for transmission to the vehicle. Further, a control system may instruct compressing the vehicle commands by the telestation processing island.

[0091] The process 500 may proceed with transmitting the vehicle commands to the vehicle using the telestation connectivity island, as at 514. For example, the telestation connectivity island may receive the encoded or compressed outputs from the telestation processing island, which outputs may also have been checked or verified by the telestation safety island. Then, the telestation connectivity island may transmit the vehicle commands or other outputs to the vehicle. Moreover, a connectivity island of the vehicle may receive the vehicle commands or other outputs from the telestation, which may then be further processed and executed for teleoperation or autonomous operation of the vehicle. Further, a control system may instruct transmitting the vehicle commands or other outputs by the telestation connectivity island to the vehicle.

[0092] The process 500 may then end, as at 516.

[0093] FIG. 6 is a schematic diagram 600 of an example centralized teleoperations center with virtualized telestations, in accordance with implementations of the present disclosure.

[0094] Generally, teleoperator stations for remote operation of vehicles may be specifically provisioned, programmed, or configured for particular vehicle types, platforms, or classes of vehicles. For example, various encoding and decoding standards or technologies may be used to compress, transmit, and decompress imaging data, other sensor data, vehicle commands, or other outputs data between teleoperator stations and vehicles. Thus, a teleoperator station may be provisioned or configured to use the same or corresponding encoding and decoding technologies as a vehicle with which it communicates for remote operation.

[0095] In addition, different vehicle manufacturers may each manufacture or develop one or more different vehicle types, platforms, or classes, which may each include different types, numbers, or configurations of cameras, sensors, drive systems or controllers, braking systems or controllers, steering systems or controllers, vehicle motion controllers, peripheral systemsor controllers, associated interfaces, and / or other components or systems. As a result, a teleoperator station may also be provisioned or configured, e.g., with specific hardware and / or software, to receive and send data to a particular or designated vehicle type, platform, or class for remote operation. Thus, multiple different teleoperator stations that are provisioned, configured, or dedicated for particular vehicle types, platforms, and classes may be needed, in order to enable remote operation of such variety of vehicles.

[0096] Further, in order to reconfigure a teleoperation station to communicate with a different vehicle type, platform, or class, various hardware and / or software of the teleoperator station may need to be changed, reconfigured, or modified to enable receipt and transfer of data and remote operation of the vehicle. In particular, software that is programmed to each teleoperator station may be specific or dedicated for a particular vehicle type, platform, or class, and in order to enable the teleoperator station to communicate with a different vehicle type, platform, or class, the software associated with the teleoperator station may need to be updated, reprogrammed, or reflashed with software that is compatible with the different or desired vehicle type, platform, or class.

[0097] As shown in FIG. 6, an example centralized teleoperations center 640, which is also referred to herein as a virtualization and streaming server (VSS), with multiple virtualized telestations 610 having associated embedded PCs (personal computers) 611 may be in communication with one or more vehicles 102 via a network 105. The vehicles 102 may include any and all of the features of vehicles described herein at least with respect to FIGs. 1 and 2, e.g., one or more cameras 103 or other sensors, a vehicle motion controller 104, and a vehicle teleoperation system 115.

[0098] In example embodiments, the VSS 640 may communicate with multiple vehicles 102 and transmit and receive data to and from the vehicles to enable remote operations. In some examples, the VSS 640 may be located physically near or at a location with one or more virtualized telestations 610. In other examples, the VSS 640 may be separate or remote from one or more virtualized telestations 610, e.g., similar to a cloud computing server or data center. In some examples, the VSS 640 may comprise one or more data centers that are positioned nearby or within regions in which multiple virtualized telestations 610 may operate and communicate with the VSS 640.

[0099] In addition, the VS S 640 may communicate with the one or more virtualized telestations 610 via various connection or communication methods, standards, or technologies. For example, the communication methods may comprise wired or fiber connections, ethemet connections, the Internet, wireless communication protocols, or other types of communication methods. In some examples, wired, fiber, ethernet, or similar connections may be preferred, in order to reduce or minimize latency associated with transfer of data between the VSS 640 and one or more virtualized telestations 610.

[0100] In example embodiments, the VSS 640 may comprise one or more transcoders 642 and one or more server class GPUs or GPU clusters 644. By utilizing or leveraging multiple different types of transcoders 642, the VSS 640 may be able to communicate with multiple different vehicle types, platforms, or classes. In this manner, the VSS 640 may receive sensor data from various different vehicles, decode or decompress the sensor data using appropriate or corresponding transcoders 642, and process the sensor data using the GPU clusters 644 for presentation by virtualized telestations 610 and embedded PCs 611. Further, the VSS 640 may receive vehicle commands or other outputs from the virtualized telestations 610 and embedded PCs 611, process or verify the vehicle commands or outputs using the GPU clusters 644 and / or a safety island as further described herein, and then encode or compress the vehicle commands or outputs using appropriate or corresponding transcoders 642 for transmission to the different vehicles.

[0101] Thus, the VSS 640 may select the appropriate transcoder 642 to process and transmit various data, according to the data types received from and associated with different vehicles. By incorporating and utilizing multiple different types of encoding and decoding standards or technologies as part of the VSS 640, individual virtualized telestations 610 and embedded PCs 611 may not need to be provisioned, reconfigured, reprogrammed, or reflashed to communicate with different vehicle types, platforms, or classes. Moreover, the GPU clusters 644 of the VSS 640 may perform substantially all processing of the sensor data received from vehicles for presentation by the virtualized telestations 610, and the GPU clusters 644 of the VSS 640 may also perform substantially all processing of vehicle commands and other outputs from the virtualized telestations 610 for transfer and execution by the vehicles.

[0102] Further, by performing substantially all processing of sensor data, as well as vehicle commands and other outputs, using the GPU clusters 644 of the VSS 640, thevirtualized telestations 610 and embedded PCs 611 may operate as virtual machines. For example, the embedded PCs 611 may comprise consumer-grade or off-the-shelf computing hardware that is configured to run one or more virtual machines or terminals as generated by and received from the VSS 640. In this manner, the virtualized telestations 610 and embedded PCs 611 may operate as a virtual terminals or machines that receive and present imaging data and other sensor data received from the VSS 640. In addition, the virtualized telestations 610 and embedded PCs 611 may receive user inputs or data that comprise vehicle commands or other outputs to enable remote operation of vehicles, and the various vehicle commands and other outputs may be transmitted to the VSS 640 for processing and transmission to respective vehicles.

[0103] For example, for sensor data received from a particular vehicle type, platform, or class, the VSS 640 may process the sensor data, and generate a virtual instance or terminal for remote operation of the particular vehicle. Then, the sensor data and software configuration of the virtual instance or terminal may be transmitted to a selected embedded PC 611 and associated virtualized telestation 610. The embedded PC 611 may run or execute the virtual instance or terminal and present the sensor data, e.g., imaging data or other sensor data, to facilitate remote operation by a teleoperator at the virtualized telestation 610. In some examples, the software configuration for a virtual instance or terminal may substantially emulate or replicate various control or user interface elements associated with the vehicle, e.g., steering wheel, brake, throttle, transmission selector, parking brake, various peripherals, infotainment systems, instrument gauges, or other elements. Using various user interfaces or input elements of the virtualized telestation 610, e.g., steering wheel, brake, throttle, or other control elements, various vehicle commands or other outputs may be received from the teleoperator of the virtualized telestation 610. The embedded PC 611 may receive such vehicle commands and other outputs from the virtualized telestation 610, and transmit or provide the various outputs to the VSS 640 for processing, verification, and transmission to the particular vehicle.

[0104] By utilizing the virtualized telestations 610 and embedded PCs 611 as virtual machines or terminals, the hardware and software associated therewith may be simplified and minimized. In addition, individual telestations 610 and embedded PCs 611 may be utilized to provide vehicle commands and other outputs for vehicles of different types, platforms, orclasses, without having to reconfigure, reprogram, or reflash the hardware and / or software of individual telestations 610 and embedded PCs 611 to enable appropriate communication.

[0105] In some example embodiments, the VSS 640 may comprise multiple different types of transcoders 642 that can receive and transmit data to and from multiple different vehicle types, platforms, or classes. In additional example embodiments, the VSS 640 may comprise multiple server class GPUs or GPU clusters 644, and individual GPUs or GPU clusters 644 may be configured to generate, configure, or run multiple different virtual instances or terminals to be presented via virtualized telestations 610 and embedded PCs 611, e.g., ten, twenty, thirty, fifty, or other numbers of different virtual instances or terminals. In this manner, the VSS 640 may be specialized on-demand for various different combinations of data types and vehicle types, platforms, or classes.

[0106] Furthermore, the virtualized telestations 610 and embedded PCs 611 may comprise consumer-grade, off-the-shelf, and / or general purpose computing hardware that can present various imaging data or other sensor data received from the VSS 640. In addition, individual virtualized telestations 610 and embedded PCs 611 may receive virtual instances or terminals and associated software configurations from the VSS 640, in order to present sensor data associated with particular vehicle types, platforms, or classes, and to receive vehicle commands or other outputs for such vehicle types, platforms, or classes. In this manner, the virtualized telestations 610 and embedded PCs 611 may be generalized and / or substantially interchangeable with each other, as well as being configured to receive on-demand various software configurations and associated virtual instances or terminals for multiple different vehicle types, platforms, or classes.

[0107] FIG. 7 is another schematic diagram 700 of an example centralized teleoperations center with virtualized telestations, in accordance with implementations of the present disclosure.

[0108] As shown in FIG. 7, a vehicle 102 may comprise one or more cameras or other vehicle sensors 103, a vehicle teleoperation system 115, and a vehicle motion controller 104, as described herein at least with respect to FIGs. 1 and 2. The various sensor data captured by the sensors 103 may be received by the vehicle teleoperation system 115 and at least partially processed for transmission to the VSS 640. For example, the basic or limited processing onboard the vehicle may comprise encoding or compressing the various sensordata for transmission via the network 105. Then, the vehicle teleoperation system 115 may transmit the sensor data via the network 105 to the VS S 640 for further processing.

[0109] Upon receipt of sensor data from a vehicle 102, the VSS 640 may perform various processing in order to transmit data for presentation by a virtualized telestation 610. The sensor data may comprise imaging data, audio data, time-of-flight data, and / or other sensor data. For example, based on a data type, formatting, and / or encoding associated with the sensor data, the VSS 640 may select an appropriate or corresponding transcoder to decode or decompress the sensor data.

[0110] Then, the VSS 640 may process the sensor data using various imaging data and other sensor data processing applications, models, or techniques. In some examples, imaging data may be processed using various ML models, Al models, deep learning models, neural networks, or other image processing models or algorithms. The various processing of the sensor data may comprise image segmentation, feature or object detection, object, obstacle, or path identification, potential path generation, desired path selection, potential or suggested vehicle commands generation, and / or other sensor data processing.[OHl] In additional examples, the VSS 640 may further generate or develop one or more overlays, layers, masks, or other augmented reality (AR) indications to be added or overlaid onto imaging data or other sensor data. For example, the overlays or AR indications may comprise object identifications, object or vehicle outlines or bounding boxes, pathway or lane markers or boundaries, projected or suggested path indications, and / or various other overlaid or supplemental data or information.

[0112] The VSS 640 may further perform rendering of the imaging data and other sensor data to be presented by a virtualized telestation 610. For example, the VSS 640 may generate or render imaging data, e.g., based on the processed sensor data and / or the overlays or AR indications, that can be presented to a teleoperator 613 via a camera view 612 of a virtualized telestation 610. In addition, the VSS 640 may generate or develop a virtual instance or terminal and associated software configuration for presentation by the virtualized telestation 610. As described herein, the virtual instance or terminal may be customized or specialized to present the rendered imaging and other sensor data, and to receive vehicle commands and other outputs based on a specific vehicle type, platform, or class. In some examples, the software configuration for a virtual instance or terminal may substantially emulate orreplicate various control or user interface elements associated with the vehicle, e.g., steering wheel, brake, throttle, transmission selector, parking brake, various peripherals, infotainment systems, instrument gauges, or other elements.

[0113] Then, the VSS 640 may encode or compress the rendered imaging and other sensor data, as well as the software configuration for the generated virtual instance or terminal, for transmission to the virtualized telestation 610, e.g., via ethernet, fiber, or wired network technologies or via the Internet or wireless network technologies 605. In some examples in which the VSS 640 and the virtualized telestation 610 are connected via ethernet, fiber, or other wired connections, the rendered imaging data, other sensor data, and software configuration may not be encoded or compressed. Instead, the rendered imaging data, other sensor data, and software configuration may be transmitted directly to the virtualized telestation 610, which may further reduce or minimize any latency associated with data transmission between the VSS 640 and the virtualized telestation 610.

[0114] At the virtualized telestation 610, the embedded PC 611 may receive the software configuration of the virtual instance or terminal, and may run or execute the software configuration such that the virtualized telestation 610 may receive vehicle commands or other outputs for the specific vehicle type, platform, or class. In addition, the embedded PC 611 may cause presentation of the rendered imaging data and other sensor data, e.g., via the camera view 612, to the teleoperator or user 613, so that the teleoperator may perform remote operation of the vehicle 102. During presentation of the imaging and other sensor data using the software configuration of the virtual instance or terminal, various vehicle commands, other outputs, and user inputs 614 may be received from the teleoperator 613, e.g., steering, braking, or throttle commands, and / or commands associated with various peripheral components or systems.

[0115] Then, the various vehicle commands and other outputs received by the embedded PC 611 of the virtualized telestation 610 may be transmitted via the ethernet or internet 605 to the VSS 640. In some examples, the vehicle commands and other outputs may be encoded or compressed prior to transmission to the VSS 640, whereas in other examples, the vehicle commands and other outputs may be transmitted directly, e.g., via wired connections, without any encoding or compression.

[0116] Upon receipt of the vehicle commands and other outputs, the VSS 640 may process the commands and outputs. For example, the VSS 640 may comprise a safety island that checks or verifies the vehicle commands and other outputs for various aspects or attributes, e.g., accuracy, reliability, integrity, latency, or other aspects. In some examples, the safety island may ensure that vehicle commands are not erratic or unexpected, within upper or lower limits, generally consistent with prior commands or known maneuvers, and / or otherwise appropriate for safe operation of the vehicle. Various other aspects or attributes may be checked or verified to ensure safe operation of the vehicle responsive to the vehicle commands or other outputs.

[0117] The VSS 640 may then encode or compress the vehicle commands and other outputs for transmission to the vehicle 102 via the network 105. Upon receipt of the vehicle commands and other outputs, the vehicle teleoperation system 115 may decode or decompress the vehicle commands and other outputs, and may instruct the vehicle motion controller 104 based on the vehicle commands and other outputs. Further, a safety island of the vehicle teleoperation system 115 may also check or verify the vehicle commands and other outputs for various aspects or attributes, e.g., accuracy, reliability, integrity, latency, or other aspects. Various errors and / or latency may result from the encoding, transmission, and decoding of the vehicle commands and other outputs from the VSS 640 to the vehicle 102, and the safety island may further ensure that the received data is still appropriate for safe operation of the vehicle.

[0118] As described herein, in example embodiments, the VSS 640 may comprise multiple different types of transcoders that can receive and transmit data to and from multiple different vehicle types, platforms, or classes. In addition, the VSS 640 may comprise multiple server class GPUs or GPU clusters, and individual GPUs or GPU clusters may be configured to generate, configure, or run multiple different virtual instances or terminals to be presented via virtualized telestations 610 and embedded PCs 611, e.g., ten, twenty, thirty, fifty, or other numbers of different virtual instances or terminals. In this manner, the VSS 640 may be specialized on-demand for various different combinations of data types and vehicle types, platforms, or classes.

[0119] As also described herein, in example embodiments, the virtualized telestations 610 and embedded PCs 611 may comprise consumer-grade, off-the-shelf, and / or general purpose computing hardware that can present various imaging data or other sensor data received fromthe VSS 640. In addition, individual virtualized telestations 610 and embedded PCs 611 may receive virtual instances or terminals and associated software configurations from the VSS 640, in order to present sensor data associated with particular vehicle types, platforms, or classes, and to receive vehicle commands or other outputs for such vehicle types, platforms, or classes. In this manner, the virtualized telestations 610 and embedded PCs 611 may be generalized and / or substantially interchangeable with each other, as well as being configured to receive on-demand various software configurations and associated virtual instances or terminals for multiple different vehicle types, platforms, or classes.

[0120] FIG. 8 is a flow diagram illustrating an example virtualized telestation operation process 800, in accordance with implementations of the present disclosure.

[0121] The process 800 may begin by receiving imaging data captured by vehicle cameras using a virtualization and streaming server (VSS), as at 802. For example, one or more cameras or other sensors onboard a vehicle may capture sensor data, and a processing island of the vehicle may process, e.g., encode or compress, the sensor data for transmission to the VSS using a connectivity island of the vehicle. Then, the VSS may receive the imaging and other sensor data from the vehicle. Further, a control system may instruct receiving the imaging and sensor data by the VSS.

[0122] The process 800 may continue by decompressing the imaging data using the VSS, as at 804. For example, one or more GPUs or GPU clusters, e.g., associated with the VSS, may receive the imaging and other sensor data from the vehicle. Then, the VSS may process, e.g., decode or decompress, the sensor data for further processing. Further, a control system may instruct decompressing the imaging and sensor data by the VSS.

[0123] The process 800 may proceed by processing the imaging data using GPU clusters of the VSS, as at 806. For example, the VSS may comprise one or more GPU clusters having various imaging data or sensor data processing algorithms and / or computational models, e.g., ML models, Al models, or other models, algorithms, or processing techniques. The GPU clusters may receive and process the imaging and sensor data, which may generate various outputs such as object or feature identifications, potential paths, selected paths, potential or suggested vehicle commands, or other outputs. In addition, the GPU clusters may generate or develop various overlays or AR indications to be overlaid or applied to the processed imaging or sensor data. Moreover, the GPU clusters may render the processed imaging and sensordata, as well as generate a virtual instance or terminal and associated software configuration, for presentation at a virtualized telestation. Further, a control system may instruct processing the imaging and sensor data using GPU clusters of the VS S.

[0124] The process 800 may continue to compress the processed imaging data using the VSS, as at 808. For example, the GPU clusters of the VSS may process, e.g., encode or compress, the rendered imaging and sensor data, as well as the software configuration of the virtual instance or terminal, for transmission to a virtualized telestation and embedded PC. Further, a control system may instruct compressing the various data by the VSS. In some alternative embodiments, the VSS may have a direct, fiber, or wired connection to the virtualized telestation, such that the rendered imaging data, sensor data, and software configuration may not be compressed prior to transmission to the virtualized telestation.

[0125] The process 800 may proceed to transmit the imaging data to the virtualized telestation, as at 810. For example, the VSS may transmit the imaging data, sensor data, and software configuration to the virtualized telestation and associated embedded PC. In some examples, the VSS and virtualized telestation may have a direct, fiber, or wired connection therebetween, whereas in other examples, the VSS and virtualized telestation may have an Internet, indirect, or wireless connection therebetween. Further, a control system may instruct transmitting the imaging data, sensor data, and software configuration from the VSS to the virtualized telestation.

[0126] The process 800 may continue with decompressing and presenting the imaging data using the virtualized telestation, as at 812. For example, the embedded PC of the virtualized telestation may run or execute the software configuration for the virtual instance or terminal, in order to present the imaging data and other sensor data via various displays, monitors, speakers, or other output devices associated with the virtualized telestation. As described herein, the virtual instance or terminal may substantially emulate or replicate various control or user interface elements associated with the vehicle, e.g., steering wheel, brake, throttle, transmission selector, parking brake, various peripherals, infotainment systems, instrument gauges, or other elements, while also presenting the rendered imaging data and other sensor data to a teleoperator at the virtualized telestation. Further, a control system may instruct decompressing and presenting the imaging data, sensor data, and software configuration using the virtualized telestation.

[0127] The process 800 may continue to receive vehicle commands from the virtualized telestation, as at 814. For example, based on presentation of the imaging and sensor data using the software configuration for the virtual instance or terminal, the teleoperator at the virtualized telestation may provide various user inputs via user interfaces or input elements, e.g., steering wheel, brake, throttle, other peripheral controls, or other input elements. The various user inputs from the teleoperator may correspond to various vehicle commands and / or other outputs to enable remote operation of the vehicle. In addition, the vehicle commands and other outputs may be encoded or compressed for transmission from the virtualized telestation to the VS S. Further, a control system may instruct receiving vehicle commands from the virtualized telestation. In some alternative embodiments, the virtualized telestation may have a direct, fiber, or wired connection to the VSS, such that the vehicle commands and other outputs may not be compressed prior to transmission to the VSS.

[0128] The process 800 may proceed to process and verify the vehicle commands using a safety island, as at 816. For example, the vehicle commands or other outputs may be received by the safety island of the VSS, and processed to verify various aspects of the outputs, e.g., for accuracy, reliability, integrity, latency, or other aspects. The safety island may ensure that any commands or instructions provided to the vehicle are checked or verified to ensure safe operation of the vehicle. Further, a control system may instruct processing the vehicle commands by a safety island of the VSS.

[0129] The process 800 may continue with transmitting the vehicle commands to the vehicle, as at 818. For example, the VSS may process, e.g., encode or compress, the vehicle commands or other outputs for transmission to the vehicle. Then, the VSS may transmit the vehicle commands or other outputs to the vehicle. Moreover, a connectivity island of the vehicle may receive the vehicle commands or other outputs from the VSS, which may then be further processed and executed for remote operation of the vehicle. Further, a control system may instruct transmitting the vehicle commands or other outputs by the VSS to the vehicle.

[0130] The process 800 may then end, as at 820.

[0131] It should be understood that, unless otherwise explicitly or implicitly indicated herein, any of the features, characteristics, alternatives or modifications described regarding a particular implementation herein may also be applied, used, or incorporated with any other implementation described herein, and that the drawings and detailed description of thepresent disclosure are intended to cover all modifications, equivalents and alternatives to the various implementations as defined by the appended claims. Moreover, with respect to the one or more methods or processes of the present disclosure described herein, including but not limited to the flow charts shown in FIGs. 5 and 8, orders in which such methods or processes are presented are not intended to be construed as any limitation on the claimed inventions, and any number of the method or process steps or boxes described herein can be omitted, reordered, or combined in any order and / or in parallel to implement the methods or processes described herein. Also, the drawings herein are not drawn to scale.

[0132] Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey in a permissive manner that certain implementations could include, or have the potential to include, but do not mandate or require, certain features, elements and / or steps. In a similar manner, terms such as “include,” “including” and “includes” are generally intended to mean “including, but not limited to.” Thus, such conditional language is not generally intended to imply that features, elements and / or steps are in any way required for one or more implementations or that one or more implementations necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and / or steps are included or are to be performed in any particular implementation.

[0133] The elements of a method, process, or algorithm described in connection with the implementations disclosed herein can be embodied directly in hardware, in a software module stored in one or more memory devices and executed by one or more processors, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, EPROM, EEPROM, registers, a hard disk, a removable disk, a CD ROM, a DVD-ROM or any other form of non-transitory computer-readable storage medium, media, or physical computer storage known in the art. An example storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The storage medium can be volatile or nonvolatile. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0134] Disjunctive language such as the phrase “at least one of X, Y, or Z,” or “at least one of X, Y and Z,” unless specifically stated otherwise, is otherwise understood with the context as used in general to present that an item, term, etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is not generally intended to, and should not, imply that certain implementations require at least one of X, at least one of Y, or at least one of Z to each be present.

[0135] Unless otherwise explicitly stated, articles such as “a” or “an” should generally be interpreted to include one or more described items. Accordingly, phrases such as “a device configured to” are intended to include one or more recited devices. Such one or more recited devices can also be collectively configured to carry out the stated recitations. For example, “a processor configured to carry out recitations A, B and C” can include a first processor configured to carry out recitation A working in conjunction with a second processor configured to carry out recitations B and C.

[0136] Language of degree used herein, such as the terms “about,” “approximately,” “generally,” “nearly” or “substantially” as used herein, represent a value, amount, or characteristic close to the stated value, amount, or characteristic that still performs a desired function or achieves a desired result. For example, the terms “about,” “approximately,” “generally,” “nearly” or “substantially” may refer to an amount that is within less than 10% of, within less than 5% of, within less than 1% of, within less than 0.1% of, and within less than 0.01% of the stated amount.

[0137] Example embodiments described herein may comprise a system for autonomous vehicle operation, comprising an autonomous vehicle comprising at least one camera, a vehicle teleoperation system, and a vehicle motion controller in communication with each other, the vehicle teleoperation system including a safety island to verify vehicle commands; and a telestation comprising a processing island having at least one graphics processing unit (GPU) cluster, the telestation being in communication with the autonomous vehicle via a network; wherein the vehicle teleoperation system may be configured to receive and encode imaging data captured by the at least one camera, and to transmit the imaging data to the telestation; and wherein the GPU cluster may be configured to execute at least one computational model upon the imaging data received from the autonomous vehicle, and to generate vehicle commands based on the imaging data.

[0138] Optionally, the telestation may further comprise a connectivity island configured to receive the imaging data from the autonomous vehicle, and to transmit the vehicle commands to the autonomous vehicle. Optionally, the processing island may be configured to decode the imaging data that is received from the autonomous vehicle. Optionally, the processing island may be configured to encode the vehicle commands for transmission to the autonomous vehicle. Optionally, the telestation may further comprise a telestation safety island configured to verify the vehicle commands prior to transmission to the autonomous vehicle.

[0139] Example embodiments described herein may comprise a system, comprising a telestation associated with a processing island having a graphics processing unit (GPU) cluster, the telestation being in communication with a vehicle via a network; wherein the telestation may be configured to receive sensor data from the vehicle, the vehicle being configured to capture and encode the sensor data; and wherein the GPU cluster may be configured to execute a computational model based on the sensor data to generate vehicle commands.

[0140] Optionally, the processing island may be integrated into the telestation; and the system may comprise a plurality of telestations that are in communication with a plurality of vehicles via the network. Optionally, the processing island may be integrated into a teleoperations center that is in communication with the telestation via a wired or fiber connection; and the system may comprise a plurality of telestations in communication with the teleoperations center. Optionally, the processing island may be configured to decode the sensor data received from the vehicle, the sensor data being encoded prior to transmission to the telestation by the vehicle. Optionally, the sensor data may comprise at least one of imaging data, audio data, or time-of-flight data. Optionally, the GPU cluster may comprise a plurality of GPU clusters, individual GPU clusters configured to execute a plurality of computational models based on the sensor data. Optionally, the computational model may comprise at least one of a machine learning (ML) model or an artificial intelligence (Al) model. Optionally, the computational model may be configured to perform at least one of image segmentation, feature or object detection, object, obstacle, or path identification, potential path generation, desired path selection, or vehicle commands generation.Optionally, an output of the computational model may comprise at least one of object or obstacle identifications, object or obstacle locations, one or more potential paths, a selectedpath, or one or more vehicle commands associated with respective paths. Optionally, the processing island may be configured to encode the vehicle commands for transmission to the vehicle; and the telestation may comprise a safety island that is configured to verify the vehicle commands prior to transmission to the vehicle.

[0141] Example embodiments described herein may comprise a method, comprising receiving, by a processing island associated with a telestation, sensor data from a vehicle via a network; decoding, by the processing island, the sensor data received from the vehicle, the sensor data being captured and encoded by the vehicle prior to transmission to the telestation; and executing, by a GPU cluster of the processing island, a computational model based on the sensor data to generate vehicle commands.

[0142] Optionally, the method may further comprise receiving, by a connectivity island of the telestation, the sensor data from the vehicle; and transmitting, by the connectivity island, the sensor data to the processing island. Optionally, the method may further comprise processing, by a safety island of the telestation, the vehicle commands to verify at least one accuracy, reliability, integrity, or latency associated with the vehicle commands. Optionally, the method may further comprise encoding, by the processing island, the vehicle commands for transmission to the vehicle. Optionally, the method may further comprise transmitting, by a connectivity island of the telestation, the vehicle commands to the vehicle.

[0143] Example embodiments described herein may comprise a system for remote vehicle operation, comprising a vehicle comprising at least one camera, a vehicle teleoperation system, and a vehicle motion controller in communication with each other, the vehicle teleoperation system including a safety island to verify vehicle commands; a centralized teleoperations center having at least one graphics processing unit (GPU) cluster, the centralized teleoperations center being in communication with the vehicle via a network; and a plurality of virtualized telestations in communication with the centralized teleoperations center; wherein the vehicle teleoperation system may be configured to receive and encode imaging data captured by the at least one camera, and to transmit the imaging data to the centralized teleoperations center; wherein the GPU cluster of the centralized teleoperations center may be configured to process the imaging data for presentation via a virtualized telestation of the plurality of virtualized telestations; and wherein the virtualized telestation may be configured to present the imaging data to a teleoperator, and to receive vehicle commands from the teleoperator.

[0144] Optionally, the centralized teleoperations center may be remote from at least some of the plurality of virtualized telestations. Optionally, the plurality of virtualized telestations may communicate with the centralized teleoperations center via respective wired or fiber connections. Optionally, the centralized teleoperations center may be in communication with a plurality of vehicles via the network; and the centralized teleoperations center may comprise a plurality of transcoders configured to encode and decode respective imaging data received from the plurality of vehicles. Optionally, the GPU cluster of the centralized teleoperations center may be further configured to generate a software configuration for a virtual terminal based on a type, platform, or class of the vehicle; and the virtualized telestation may be configured to present the imaging data and to receive the vehicle commands using the software configuration that replicates control elements of the vehicle.

[0145] Example embodiments described herein may comprise a system, comprising a centralized teleoperations center having at least one graphics processing unit (GPU) cluster, the centralized teleoperations center being in communication with a vehicle via a network; and a virtualized telestation in communication with the centralized teleoperations center; wherein the GPU cluster of the centralized teleoperations center may be configured to receive and process sensor data for presentation via the virtualized telestation; and wherein the virtualized telestation may be configured to present the sensor data to a teleoperator, and to receive vehicle commands from the teleoperator.

[0146] Optionally, the centralized teleoperations center may comprise a data center that is remote from the virtualized telestation. Optionally, the vehicle may comprise one of a plurality of vehicles that encode sensor data using respective ones of a plurality of encoding standards; and the centralized teleoperations center may comprise a plurality of transcoders that correspond to the plurality of encoding standards. Optionally, the centralized teleoperations center may be configured to select a transcoder of the plurality of transcoders to decode the sensor data based on an encoding standard applied to the sensor data that is received from the vehicle. Optionally, the GPU cluster may be configured to decode, process, and render the sensor data for presentation by the virtualized telestation. Optionally, the virtualized telestation may comprise one of a plurality of virtualized telestations; and the GPU cluster may be configured to generate a plurality of virtual terminals for presentation of the sensor data by the plurality of virtualized telestations. Optionally, the plurality of vehicles may comprise respective types, platforms, or classes of vehicles; and the plurality ofvirtual terminals may comprise respective software configurations that are generated to replicate one or more control or user interface elements of respective vehicles. Optionally, the plurality of virtualized telestations may be configured to receive and present the sensor data using respective software configurations that are associated with respective vehicles. Optionally, the GPU cluster may be configured to receive the vehicle commands; and a safety island of the GPU cluster may be configured to verify the vehicle commands prior to transmission to the vehicle. Optionally, the GPU cluster may be configured to encode and transmit the vehicle commands to the vehicle.

[0147] Example embodiments described herein may comprise a method, comprising receiving, by a centralized teleoperations center, sensor data from a vehicle via a network; processing, by a GPU cluster of the centralized teleoperations center, the sensor data for presentation by a virtualized telestation; generating, by the GPU cluster, a virtual terminal associated with the sensor data based on a type, platform, or class of the vehicle; and transmitting, by the centralized teleoperations center, the sensor data and the virtual terminal for presentation by the virtualized telestation.

[0148] Optionally, the sensor data may be encoded using an encoding standard by the vehicle prior to transmission to the centralized teleoperations center; and the method may further comprise selecting, by the GPU cluster from a plurality of transcoders, a transcoder that corresponds to the encoding standard; and decoding, by the GPU cluster, the sensor data received from the vehicle using the transcoder. Optionally, the virtual terminal may comprise one of a plurality of virtual terminals that are associated with respective types, platforms, or classes of vehicles. Optionally, the centralized teleoperations center may be configured to receive respective sensor data from a plurality of vehicles via the network. Optionally, the centralized teleoperations center may be configured to transmit respective sensor data and respective virtual terminals to a plurality of virtualized telestations for presentation.

[0149] Although the invention has been described and illustrated with respect to illustrative implementations thereof, the foregoing and various other additions and omissions may be made therein and thereto without departing from the spirit and scope of the present disclosure.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. A system for autonomous vehicle operation, comprising:an autonomous vehicle comprising at least one camera, a vehicle teleoperation system, and a vehicle motion controller in communication with each other, the vehicle teleoperation system including a safety island to verify vehicle commands; anda telestation comprising a processing island having at least one graphics processing unit (GPU) cluster, the telestation being in communication with the autonomous vehicle via a network;wherein the vehicle teleoperation system is configured to receive and encode imaging data captured by the at least one camera, and to transmit the imaging data to the telestation; andwherein the GPU cluster is configured to execute at least one computational model upon the imaging data received from the autonomous vehicle, and to generate vehicle commands based on the imaging data.

2. The system of claim 1, wherein the telestation further comprises a connectivity island configured to receive the imaging data from the autonomous vehicle, and to transmit the vehicle commands to the autonomous vehicle.

3. The system of claim 1 or claim 2, wherein the processing island is configured to decode the imaging data that is received from the autonomous vehicle.

4. The system of any one of claims 1, 2, or 3, wherein the processing island is configured to encode the vehicle commands for transmission to the autonomous vehicle.

5. The system of any one of claims 1, 2, 3, or 4, wherein the telestation further comprises a telestation safety island configured to verify the vehicle commands prior to transmission to the autonomous vehicle.

6. A system, comprising:a telestation associated with a processing island having a graphics processing unit (GPU) cluster, the telestation being in communication with a vehicle via a network;wherein the telestation is configured to receive sensor data from the vehicle, the vehicle being configured to capture and encode the sensor data; andwherein the GPU cluster is configured to execute a computational model based on the sensor data to generate vehicle commands.

7. The system of claim 6, wherein the processing island is integrated into the telestation; andwherein the system comprises a plurality of telestations that are in communication with a plurality of vehicles via the network.

8. The system of claim 6, wherein the processing island is integrated into a teleoperations center that is in communication with the telestation via a wired or fiber connection; andwherein the system comprises a plurality of telestations in communication with the teleoperations center.

9. The system of claim 6, wherein the processing island is configured to decode the sensor data received from the vehicle, the sensor data being encoded prior to transmission to the telestation by the vehicle.

10. The system of claim 6 or claim 9, wherein the sensor data comprises at least one of imaging data, audio data, or time-of-flight data.

11. The system of any one of claims 6, 9, or 10, wherein the GPU cluster comprises a plurality of GPU clusters, individual GPU clusters configured to execute a plurality of computational models based on the sensor data.

12. The system of any one of claims 6, 9, 10, or 11, wherein the computational model comprises at least one of a machine learning (ML) model or an artificial intelligence (Al) model.

13. The system of any one of claims 6, 9, 10, 11, or 12, wherein the computational model is configured to perform at least one of image segmentation, feature or object detection, object, obstacle, or path identification, potential path generation, desired path selection, or vehicle commands generation.

14. The system of any one of claims 6, 9, 10, 11, 12, or 13, wherein an output of the computational model comprises at least one of object or obstacle identifications, object or obstacle locations, one or more potential paths, a selected path, or one or more vehicle commands associated with respective paths.

15. The system of any one of claims 6, 9, 10, 11, 12, 13, or 14, wherein the processing island is configured to encode the vehicle commands for transmission to the vehicle; and wherein the telestation comprises a safety island that is configured to verify the vehicle commands prior to transmission to the vehicle.

Images