Machine Learning Data Rule Generation for Network Transfers

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Establishing optimal data rules for new datasets is a rate-limiting step, especially for large institutions handling vast data, as existing data rules become outdated due to changes in applications and data integration requirements.

Innovation Solution

A method utilizing machine learning engines to identify similar historical datasets and leverage their data rules for optimizing data transfers, involving querying a global dataset to match patterns and generate or optimize data rules for new datasets.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data rules are established manually or through traditional methods for new datasets, then the data transfer process can be completed, but the establishment process is time-consuming and becomes a rate-limiting step

Engineering Contradiction:

Improvedata transfer efficiencyVSAvoidtime to establish data rules

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system pre-establishes data rules during the ingestion phase when data arrives at the data lake, rather than waiting until data transfer is needed. Machine learning models analyze the ingested data and generate appropriate data rules in advance, so that when transfer operations are required, optimized rules are already available, eliminating the time-consuming rule establishment step from the critical transfer path

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system enables data rules to be automatically generated and optimized through machine learning without requiring manual intervention. The ML models self-adjust and self-optimize data rules based on patterns learned from historical data and transfer outcomes, allowing the system to service itself rather than relying on human experts to create and maintain data rules for each new dataset

Inventive Principle:

Principle #25Self-service

2Productivity

If data rules are optimized for specific applications and datasets, then data transfer efficiency improves, but the rules become outdated when applications are merged or replaced

Engineering Contradiction:

Improvedata transfer efficiencyVSAvoidadaptability to application changes

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The system transitions from static, application-specific data rules to dynamic, adaptive rules that automatically adjust to changing conditions. Machine learning models continuously monitor data patterns and transfer performance, automatically updating data rules to reflect current application states and data characteristics, ensuring rules remain optimized even as applications are merged or replaced

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system creates universal data rules through machine learning that can apply across multiple applications and datasets rather than creating separate rules for each specific application. The ML models learn generalizable patterns that work across different data types and application contexts, making the data rules more versatile and less susceptible to becoming outdated when individual applications change

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Loss of time

If machine learning models are trained on historical data to optimize data rules, then the establishment of data rules for new datasets becomes faster, but the system requires access to and processing of large volumes of historical data

Engineering Contradiction:

Improvetime to establish data rulesVSAvoidvolume of historical data required

Core Design Contradiction:

Loss of timeVSQuantity of substance

Solution Approach 1:

The system extracts only the essential features and patterns from historical data that are relevant for generating data rules, rather than processing entire datasets. Machine learning models identify and extract key characteristics such as data type patterns, transfer frequency, and performance metrics, using only these extracted features for training, thereby reducing the effective volume of historical data that must be processed while maintaining rule optimization quality

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS12182129B2Methods and systems for data leveraging

Publication Date: 2024.12.31 BANK OF AMERICA CORP

AI summary

A method for facilitating transfer of a new dataset across a network may include the following steps: (a) querying the global dataset via a computer processor to identify an existing dataset having data points corresponding to the new dataset; (b) identifying historical data rules previously used for a set of data transfers relating to the existing dataset; and (c) using the historical data rules to assist a machine learning engine in generating data rules for use with a data transfer relating to the new dataset.