Schema-Routed Data Pipelines for Low-Latency Vehicle Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems experience latency issues due to inefficient data transfer and processing, particularly when moving data between cloud VMs and internal storage, which hinders fast access to vehicle data for applications.
Innovation Solution
A data pipeline system that utilizes a generic data pipeline with schema validation and parallelization to ensure efficient, low-latency data transfer and processing, incorporating hardware abstraction layers to normalize sensor data and provide value-added insights, and a network of Lambda functions to route and process data based on schema.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is transferred from cloud VMs to internal storage through traditional data processing systems, then data storage is achieved, but latency increases and fast access is hindered
Solution Approach 1:
The data pipeline is segmented into multiple specialized pathways (real-time pathway, near-real-time pathway, batch pathway) based on data processing requirements. Each pathway is optimized for specific latency tolerances, allowing critical data to bypass complex processing steps and achieve faster access while non-critical data undergoes thorough processing.
Solution Approach 2:
Data is pre-processed and staged in intermediate storage layers before final destination. The system performs preliminary validation, schema verification, and routing decisions at the edge, preparing data in advance so that when applications need access, the data is already in an optimized state for rapid retrieval.
2Reliability
If comprehensive data processing and validation is performed, then data quality and reliability are improved, but processing time and latency increase
Solution Approach 1:
Different levels of processing quality are applied to different data streams based on their specific requirements. Critical safety-related data receives comprehensive validation and processing, while less critical telemetry data undergoes minimal processing. Each data pathway has customized processing intensity matched to its reliability requirements.
Solution Approach 2:
The system applies partial processing to some data streams and excessive processing to others. Not all data requires full validation cycles - the system performs just enough processing to meet the specific reliability needs of each data type, avoiding unnecessary processing steps that would increase latency without adding value.
3Device complexity
If a unified data processing pipeline is used for all data types, then system simplicity is maintained, but performance optimization for different data requirements is lost
Solution Approach 1:
The data pipeline is designed to be dynamic and adaptive rather than static. Routing decisions are made in real-time based on data characteristics, schema identifiers, and current system conditions. The pipeline can dynamically allocate resources and adjust processing pathways to optimize speed for different data types while maintaining a unified architectural framework.
Solution Approach 2:
A single unified pipeline architecture serves multiple functions by incorporating diverse processing pathways within one system. The same infrastructure handles real-time, near-real-time, and batch processing through configurable routing rules, maintaining structural simplicity while enabling performance optimization through multi-functional processing capabilities.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method at a data pipeline in computing system, the method including receiving data and a schema identifier associated with the data; confirming that the schema identifier is valid; and routing the data at the data pipeline based on the schema identifier. Also, a computing system having a processor; a data pipeline; and a communications subsystem, wherein the computing system is configured to: receive data and a schema identifier associated with the data; confirm, using the processor, that the schema identifier is valid; and route the data at the data pipeline based on the schema identifier.