Stratified Data Pipelines With Schema Validation for Low-Latency Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems experience latency issues due to inefficient data transfer and processing, particularly in distributed systems like vehicles, where sensors generate insights that require fast access and handling of diverse data formats across varying hardware and software configurations.
Innovation Solution
A generic data pipeline with schema validation and parallelization is implemented to efficiently route and process data, using a Hardware Abstraction Layer (HAL) and Vehicle Abstraction Layer (VAL) to normalize sensor data, and a data processing pipeline with Lambda functions to manage data routing and storage, ensuring low latency and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is transferred from cloud VMs or external data networks to internal storage, then data accessibility is improved, but latency increases
Solution Approach 1:
The patent segments the data pipeline into multiple processing stages with schema validation at each stage. Data is routed through different pipeline instances based on schema validation results, allowing parallel processing and reducing overall latency while maintaining data accessibility across the distributed system.
Solution Approach 2:
The patent introduces schema validation as an intermediary component between data transfer and data processing. This intermediary validates data schemas early in the pipeline, enabling efficient routing decisions and reducing latency by preventing invalid data from progressing through the entire transfer and processing chain.
2Adaptability or versatility
If data processing is performed on diverse sensor data formats, then adaptability is improved, but processing complexity increases
Solution Approach 1:
The patent implements a universal schema validation mechanism that handles multiple data formats through a single standardized interface. The schema validation framework is designed to work with diverse sensor data formats (JSON, CSV, binary) without requiring format-specific processing logic, thereby improving adaptability while reducing processing complexity.
Solution Approach 2:
The patent changes the parameter of data representation by enforcing schema-based standardization. Incoming diverse data formats are validated against predefined schemas that define expected structures, allowing the system to handle format diversity through parameter validation rather than complex format-specific processing logic.
3Reliability
If schema validation is performed on incoming data, then data quality is improved, but processing time increases
Solution Approach 1:
The patent performs schema validation as a preliminary action early in the data pipeline, before data is routed to various processing instances. This preliminary validation ensures data quality is established upfront, allowing subsequent processing stages to operate on already-validated data without repeating validation checks, thereby minimizing overall processing time.
Solution Approach 2:
The patent implements partial schema validation by validating only critical schema elements that are necessary for routing and processing decisions. This selective validation approach maintains data quality for essential fields while reducing the time overhead compared to complete schema validation of all data elements.
4Productivity
If data is routed based on schema identifier, then processing efficiency is improved, but system complexity increases
Solution Approach 1:
The patent creates copies of validated data with schema identifiers attached, routing these copied data instances to appropriate processing pipelines based on the schema identifier. This copying mechanism enables efficient routing decisions without requiring complex real-time analysis of the actual data content, thereby improving processing efficiency while managing system complexity through standardized data duplication.
Data Source
AI summary
A method at a data pipeline in computing system, the method including receiving data and a schema identifier associated with the data; confirming that the schema identifier is valid; and routing the data at the data pipeline based on the schema identifier. Also, a computing system having a processor; a data pipeline; and a communications subsystem, wherein the computing system is configured to: receive data and a schema identifier associated with the data; confirm, using the processor, that the schema identifier is valid; and route the data at the data pipeline based on the schema identifier.