Smart City Data Schema Matching via QoS Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for matching data schemata with inputs in smart city applications are inefficient due to reliance on manual annotation and lack of automation, leading to high deployment costs and inaccurate data mining from data lakes.
Innovation Solution
A method and system that utilize aggregate statistical information and Quality-of-Service (QoS) feedback to automatically select optimal data inputs for smart city applications, employing a trained QoS scoring function to minimize QoS error and improve predictive accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual annotation and expert confirmation are used for data schema matching, then matching accuracy is improved, but deployment cost and time consumption increase
Solution Approach 1:
The system pre-computes aggregate statistical information (mean, standard deviation, min, max, count) for all data schema concepts and stores them in advance. When a matching query is received, the global matcher can immediately compute QoS scores using these pre-computed statistics without performing expensive data analysis at query time, thus resolving the contradiction between accuracy and speed
Solution Approach 2:
Instead of manually annotating each data schema concept, the system creates a computational model (QoS scoring function) that automatically generates matching scores based on statistical properties of the data. This computational copy replaces the need for manual expert annotation, significantly reducing deployment time while maintaining matching quality through automated optimization
2Measurement precision
If more data schema concepts and statistical information are processed, then data mining accuracy is improved, but computational complexity increases
Solution Approach 1:
The system extracts only the essential statistical properties (mean, standard deviation, min, max, count) from the data schemas and uses these extracted features as input to the QoS scoring function. By taking out only the most relevant statistical information rather than processing all possible data characteristics, the system maintains high matching accuracy while significantly reducing computational complexity
Solution Approach 2:
The QoS scoring function transforms the raw statistical parameters into normalized scores through mathematical transformations (e.g., z-scores, logarithmic scaling). This parameter transformation allows the system to work with standardized values that are computationally efficient to process while still capturing the essential variations in data quality and suitability for different application inputs
3Productivity
If automated matching without manual confirmation is implemented, then deployment cost is reduced, but matching reliability decreases
Solution Approach 1:
The QoS scoring function is trained using feedback from ground-truth matches confirmed by domain experts. The system learns from these expert-validated examples to improve its automated scoring accuracy over time. This feedback mechanism allows the system to achieve high reliability through automated means by continuously learning from expert judgments rather than requiring manual confirmation for each new matching task
Solution Approach 2:
The global matcher performs self-validation by computing QoS scores and automatically selecting the best matching data schema concepts without requiring external expert intervention. The system serves itself by using the trained scoring function to evaluate and rank potential matches, achieving both high productivity through automation and maintained reliability through the quality of the trained model
Data Source
AI summary
A method for automatically selecting input for a smart city application includes determining data schema for a data lake locally stored at a local smart city platform instance and determining aggregate statistical information of data corresponding to a data schema concept. The determined data schema and aggregate statistical information are sent to a global matcher with an identifier of the application. An input matching is received, from the global matcher, for the application, wherein the input matching provides, for each respective input of the application, a respective data schema concept. Data corresponding to the respective data schema concept for each respective input of the data-driven application is provided as input to the application. The input matching is determined using a trained quality of service (QOS) scoring function trained to minimize a QoS error of aggregated QoS scores against a ground-truth QoS score computed from predictive accuracy and/or user feedback.