Dynamic Data Processing Configuration for Stream Efficiency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data processing systems face challenges in efficiently handling large streams of data due to incompatibility issues between different data structures, leading to resource-intensive processing and delays.

Innovation Solution

A system that uses a machine learning model to dynamically configure processing parameters for streaming data by identifying real-time data parameters and predicting optimal processing parameters, thereby optimizing data processing efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional data processing systems are used to handle large streams of data, then data processing can be performed, but computing resources are consumed excessively and processing delays occur

Engineering Contradiction:

Improvedata processing efficiencyVSAvoidcomputing resource consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent implements dynamic configuration of processing parameters based on real-time data characteristics. The system continuously monitors data stream properties (such as data volume, complexity, and patterns) and automatically adjusts processing parameters like batch size, processing speed, and resource allocation to optimize the balance between processing efficiency and resource consumption, resolving the contradiction between productivity and energy usage

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes processing parameters dynamically based on detected data characteristics. By modifying parameters such as processing batch size, thread count, and memory allocation according to real-time data stream properties, the system achieves efficient resource utilization while maintaining high processing throughput, thereby resolving the contradiction between productivity and energy consumption

Inventive Principle:

Principle #35Parameter changes

2Productivity

If traditional data processing systems are used to handle large streams of data, then data processing can be performed, but processing delays occur

Engineering Contradiction:

Improvedata processing throughputVSAvoidprocessing delay

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system dynamically adjusts processing parameters in real-time based on data stream characteristics and system load conditions. When data volume increases, the system automatically increases processing capacity; when data volume decreases, it reduces capacity to avoid delays. This dynamic adaptation maintains optimal processing throughput while minimizing delays, resolving the contradiction between productivity and time loss

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system incorporates feedback mechanisms that continuously monitor processing performance and data characteristics. Based on this feedback, the system automatically adjusts processing parameters to maintain optimal throughput and reduce delays. The feedback loop enables real-time optimization of the balance between processing speed and time efficiency

Inventive Principle:

Principle #23Feedback

3Adaptability or versatility

If multiple data structures from different sources are integrated, then data from multiple sources can be processed, but incompatibility issues arise leading to resource-intensive processing

Engineering Contradiction:

Improvedata source compatibilityVSAvoidprocessing complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system introduces a normalization layer that acts as an intermediary between diverse data sources and the processing engine. This normalization layer automatically transforms data from multiple incompatible structures into a unified representation, enabling versatile data source integration while keeping the processing logic simple and resource-efficient, thus resolving the contradiction between adaptability and complexity

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20250200424A1Dynamic configuration of a data processing system

Publication Date: 2025.06.19 CAPITAL ONE SERVICES LLC

AI summary

In some implementations, a data processing system may identify one or more real-time data parameters associated with a data stream. The data processing device may determine, using at least one of a machine learning model or a set of rules, and based on the real-time data parameters, a set of optimal processing parameters associated with processing the data stream. The data processing system may configure a data processing device with the set of optimal processing parameters. The data processing device may process the data stream using the data processing device and based on the set of optimal processing parameters.