Explainable Feature Pipeline With Lineage Across Cloud Data Sources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The proliferation of cloud services and execution environments poses challenges for feature engineering due to difficulties in combining data from different cloud platforms, lack of common schemas, and variations in metrics and data collection APIs, leading to inconsistent feature availability and quality.
Innovation Solution
A feature engineering data pipeline that ensures uniformity in data collection, transformation, and ingestion, with separate feature generation logic from execution environment, allowing for real-time processing and metadata generation to maintain data integrity and traceability, and supports various feature generation frequencies across multiple cloud platforms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is combined from different cloud platforms, then data volume and source diversity increase, but data consistency and schema uniformity deteriorate
Solution Approach 1:
The patent introduces a feature store as an intermediary layer between multiple cloud platforms and machine learning models. This feature store standardizes data schemas and provides a unified interface, allowing data from diverse cloud sources to be combined without compromising consistency. The feature store acts as a mediator that translates various cloud platform data formats into a common schema.
Solution Approach 2:
The patent segments the data pipeline into distinct components: data collection from cloud platforms, feature engineering, feature storage in the feature store, and model training. This segmentation allows each component to handle specific tasks with appropriate data formats, while the feature store ensures schema uniformity across the entire pipeline.
2Measurement precision
If features are engineered to improve predictive power, then model accuracy improves, but feature availability and stability deteriorate over time
Solution Approach 1:
The patent performs feature engineering in advance and stores the engineered features in a feature store before model training. This preliminary action ensures that features are pre-processed and made available independently of when models are trained, improving feature availability and stability over time.
Solution Approach 2:
The feature store provides a universal repository that serves multiple machine learning models and multiple training iterations. Once features are engineered and stored, they can be reused across different models and training runs, ensuring feature availability and stability without requiring re-engineering.
3Manufacturing precision
If manual feature engineering is performed, then feature quality improves, but processing time and manual intervention increase
Solution Approach 1:
The patent implements automated feature engineering within the data pipeline that can automatically discover, engineer, and store features without continuous manual intervention. The system self-services by automatically processing data through the feature engineering stage and populating the feature store, reducing manual effort while maintaining feature quality.
Solution Approach 2:
The patent establishes a continuous data pipeline that automatically performs feature engineering and updates the feature store. This continuous automation eliminates the need for repeated manual feature engineering for each model training iteration, significantly reducing processing time while maintaining consistent feature quality.
4Speed
If real-time feature processing is implemented, then prediction timeliness improves, but data transformation complexity increases
Solution Approach 1:
The patent performs data transformation and feature engineering in advance, storing processed features in the feature store before real-time prediction is needed. This preliminary processing reduces the complexity of real-time transformations while maintaining prediction timeliness, as the heavy lifting is done beforehand.
Solution Approach 2:
The feature store serves as an intermediary that provides pre-processed features for real-time predictions. This mediator approach simplifies real-time processing by providing ready-to-use features without requiring complex real-time transformations, thus improving prediction timeliness while managing transformation complexity.
Data Source
AI summary
A system and method are provided for feature engineering. The method may include obtaining data from a plurality of data sources. The method may also include generating, based on the data, features for training machine learning models using a single data pipeline that uses a feature generation logic that is separate from an execution logic. The method may also include generating, based on relations and/or joins in the feature generation logic, metadata for explaining the features, and generating, based on the metadata, lineage information representing a mapping of one or more features to intermediate tables and/or input tables. The method may also include displaying the lineage information representing the mapping, including allowing a user to select and/or drill down different features, intermediate tables and/or input tables. The method may also include storing the features and the metadata for the features to a feature store.