Decoupled Data Transformation Logic for ML Pipelines

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Machine learning models require frequent updates and resource-intensive maintenance due to inconsistent data processing and tied data retrieval logic, leading to inefficiencies and redundancy across different execution contexts and technologies.

Innovation Solution

A system with decoupled transformation logic units that reference needed data via metadata, allowing separate data gathering and computation, enabling reuse across various contexts and technologies, and storing logic metadata for transparency and reproducibility.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If data retrieval and transformation logic are tied together in a single script, then data can be retrieved and processed, but the logic becomes difficult to reuse across different machine learning models and execution contexts

Engineering Contradiction:

Improvereusability of transformation logicVSAvoidcomplexity of data retrieval and transformation system

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments the data retrieval and transformation logic into separate, independent components. The transformation logic is extracted as a distinct unit that can be reused across different models, while the data retrieval logic remains separate. This segmentation enables the transformation logic to be shared across multiple machine learning models without requiring code duplication.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The transformation logic is extracted from the data retrieval script and stored as a separate, reusable unit. This extracted transformation logic can be independently applied to different data sources and models, eliminating the need to rewrite transformation code when working with different datasets or models.

Inventive Principle:

Principle #2Taking out (Extraction)

2Adaptability or versatility

If data derivation logic is tied to a specific database or technology, then data can be retrieved efficiently, but the logic must be rewritten when technology changes

Engineering Contradiction:

Improveportability across technologiesVSAvoidtime required to rewrite logic when technology changes

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The patent introduces an intermediary layer between the data retrieval logic and the transformation logic. This intermediary acts as a technology-agnostic interface that allows transformation logic to be written once and executed across different databases and technologies without requiring modifications to the transformation logic itself.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The transformation logic is designed as a universal component that can work with multiple data sources and technologies. By abstracting the transformation logic from specific technology implementations, the same logic can be applied across different databases, cloud platforms, and data warehouses without rewriting.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Adaptability or versatility

If transformation logic is written for a specific execution context, then the logic can be optimized for that context, but the logic cannot be reused in other execution contexts

Engineering Contradiction:

Improvecross-context reusabilityVSAvoidease of authoring and maintaining logic

Core Design Contradiction:

Adaptability or versatilityVSEase of manufacture

Solution Approach 1:

The transformation logic is designed as a universal component that can be executed across multiple contexts including batch processing, streaming, and real-time inference. The logic is abstracted from specific execution context details, allowing the same transformation code to function in different environments without modification.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The system dynamically adapts the execution of transformation logic based on the execution context. The same transformation logic can be executed with different parameters and configurations depending on whether it's running in batch mode, streaming mode, or real-time inference, providing flexibility without requiring separate logic for each context.

Inventive Principle:

Principle #15Dynamics

4Productivity

If data retrieval scripts are copied and pasted for different models, then the same data can be retrieved, but redundancy and inconsistency increase

Engineering Contradiction:

Improvespeed of data retrievalVSAvoidconsistency of data processing logic

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

Instead of copying and pasting entire data retrieval and transformation scripts, the patent uses a centralized repository of transformation logic that can be referenced and applied to multiple models. This eliminates redundancy while maintaining consistency, as the same transformation logic is applied uniformly across all models without manual copying.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS20220269978A1Systems and methods for implementing data transformations in multiple execution contexts

Publication Date: 2022.08.25 CAPITAL ONE SERVICES LLC

AI summary

A data transformation system for implementing reproducible and consistent data transformations in multiple execution contexts (batch, streaming, etc.) where the transformation function/logic initially acts on historical raw data to produce derived data to train a machine learning model. When the model is trained and deployed to handle streaming event data, the same transformation is reused to transform streaming data into the appropriate derived data for the model scoring, and later for a refit of the model.