Smart Data Ingestion via Value Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data ingestion methods for automated and autonomous vehicles are inefficient, leading to slow access and high costs due to improper storage tiering and imprecise data categorization, especially when distinguishing between hot and cold data.
Innovation Solution
A method for ingesting data records that involves retrieving content-related descriptors to assign value classifications, allowing for the selection of appropriate transmission channels and storage solutions based on the classification, thereby optimizing bandwidth, transportation time, and storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If more storage servers and processing units are provided to handle large sums of data, then data handling capacity is improved, but costs increase significantly
Solution Approach 1:
The patent segments data into different categories (hot data, cold data, archived data) based on access patterns and importance. This segmentation allows the system to apply different storage and transmission strategies to different data types, optimizing resource allocation and reducing overall costs while maintaining high data handling capacity.
Solution Approach 2:
The patent applies local quality by assigning different value classifications to different data records based on their specific characteristics and usage patterns. Each data record receives tailored transmission and storage treatment according to its individual value assessment, rather than applying a uniform approach to all data.
2Quantity of substance
If data is moved to cheaper storage solutions based on historical usage, then storage costs are reduced, but data access speed decreases and categorization is imprecise
Solution Approach 1:
The patent performs preliminary action by assessing and classifying data value at the point of ingestion, before storage decisions are made. This upfront classification ensures that data is placed in the appropriate storage tier from the beginning, eliminating the need for later data movement and avoiding access speed penalties while still achieving cost optimization.
Solution Approach 2:
The patent replaces the mechanical system of physical data movement between storage tiers with an information-based system. By using automated value assessment and classification algorithms, the system determines optimal storage placement without requiring actual data migration, thereby maintaining fast access speeds while reducing storage costs.
3Speed
If hot data is stored on fast storage solutions, then data access speed is improved, but storage costs increase
Solution Approach 1:
The patent changes the parameter of storage speed allocation by dynamically adjusting which data receives fast storage based on real-time value assessment. Instead of statically designating all hot data for fast storage, the system continuously evaluates data value and adjusts storage allocation accordingly, optimizing the balance between access speed and storage cost.
4Loss of time
If data transmission bandwidth is increased for faster data availability, then data access time is reduced, but transmission costs increase
Solution Approach 1:
The patent applies local quality to data transmission by assigning different transmission priorities and bandwidth allocations to different data records based on their assessed value. High-value data receives preferential treatment with faster transmission channels, while lower-value data uses standard or deferred transmission, optimizing the balance between data availability time and transmission costs.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
A method, an apparatus, and a computer program for data ingestion. Content-related descriptors are retrieved. Each descriptor is associated with a value classification. Furthermore, a data record to be ingested is retrieved. A value classification is assigned to the data record based on the descriptors. The data record is then ingested in accordance with the assigned value classification.