Dynamic Metadata Extraction for Measurement Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for analyzing measurement data are time-consuming, especially when dealing with large datasets, as they require transferring data files into a data lake, followed by filtering and parsing, which consumes significant compute time, particularly in applications like Autonomous Driving and ADAS that process vast amounts of data.
Innovation Solution
A system and method for dynamically analyzing measurement data as it is received and stored, which involves extracting and generating metadata in real-time, allowing for the identification of data subsets based on predefined criteria, and storing this information directly in a data warehouse without intermediate storage, using a file ingestion logic module and a streaming module that operates on an event-based model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If measurement data is transferred to a data lake and then filtered and parsed, then data analysis can be performed, but the process consumes significant time and compute resources
Solution Approach 1:
The patent extracts and stores metadata during the data ingestion phase, before analysis is needed. This preliminary extraction of measurement data points, units, and other metadata eliminates the need for time-consuming filtering and parsing operations later, as the data is already structured and indexed for direct querying.
Solution Approach 2:
The patent extracts only the essential metadata (measurement data points, units, timestamps) from the raw measurement data during ingestion, separating this structured information from the complete raw data files. This extraction allows analysis to proceed using only the necessary metadata without requiring access to or processing of the entire raw data set.
2Reliability
If all measurement data files are transferred to a data lake, then complete data is available for analysis, but the transfer and storage process consumes significant compute time
Solution Approach 1:
The patent extracts and stores only the essential metadata (measurement data points, units, timestamps, descriptions) from the raw measurement data during the ingestion phase. This selective extraction maintains data reliability for analysis purposes while dramatically reducing the storage and transfer requirements compared to moving complete raw data files.
Solution Approach 2:
The patent segments the measurement data into two components: extracted metadata (stored in the data warehouse for efficient querying) and raw data files (retained in their original location). This segmentation allows analysis to proceed using only the metadata segment, eliminating the need to transfer and process the entire data set.
3Loss of time
If metadata is extracted and stored dynamically during data movement, then analysis time is reduced, but the data processing system becomes more complex
Solution Approach 1:
The patent implements metadata extraction as part of the standard data ingestion process, performing the extraction operation during the initial data movement phase. This preliminary action embeds the extraction logic within the existing data pipeline, avoiding the need for separate, complex post-processing systems while still achieving the time-saving benefits.
Data Source
AI summary
A method to dynamically analyze measurement data comprising measurement data sets as the measurement data is received and moved to a data warehouse. The program instructions may receive the measurement data and may extract first metadata from the measurement data. The program instructions may then extract and analyze measurement data points in the measurement data to determine if the measurement data points meet a first criteria and generate second metadata in response to determining that the measurement data points meet the first criteria. The program instructions may then provide the measurement data points, the first metadata and the second metadata to a data warehouse for storage. The analysis of the measurement data and creation of new metadata may be performed dynamically as the data is acquired and stored in the data warehouse.