ML Pipeline Data Frame Code Translation for Faster Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Data processing code in machine learning pipelines using Pandas libraries is slower than similar code written with high-performance libraries like NumPy, due to repeated computations and lack of caching and reuse of specific caching and reuse of specific caching, leading to inefficiencies in existing systems.

Innovation Solution

A system and method for acceleration of slower data processing code in machine learning pipelines by automatically generating an accelerated data processing code using hardware processors to identify and generate an accelerated data processing code.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If Pandas data frame API is used for data processing, then ease of operation is improved, but processing speed deteriorates

Engineering Contradiction:

Improveease of useVSAvoidprocessing speed

Core Design Contradiction:

Ease of operationVSSpeed

Solution Approach 1:

The system creates an accelerated copy of the Pandas data processing code by translating it into equivalent NumPy operations. The translation layer copies the functionality of Pandas operations but executes them using NumPy's optimized underlying implementations, thereby maintaining ease of use while improving processing speed.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system substitutes the Pandas mechanical processing system with a NumPy-based processing system. By replacing Pandas operations with their NumPy equivalents, the system leverages NumPy's optimized C-based implementations to achieve faster execution while preserving the high-level usability of Pandas syntax through automatic translation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Ease of operation

If Pandas data frame operations are used, then ease of operation is improved, but productivity deteriorates

Engineering Contradiction:

Improveease of useVSAvoidtemporal performance

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system changes the execution parameters of data processing operations by translating high-level Pandas operations into lower-level NumPy operations with optimized execution parameters. This parameter transformation enables faster execution while maintaining the same functional output, thereby improving productivity without sacrificing ease of operation.

Inventive Principle:

Principle #35Parameter changes

3Speed

If automatic code translation is implemented, then processing speed is improved, but device complexity increases

Engineering Contradiction:

Improveprocessing speedVSAvoidsystem complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The system introduces an intermediary translation layer between the user's Pandas code and the execution engine. This intermediary automatically translates Pandas operations into NumPy operations, providing the speed benefits of NumPy while shielding users from complexity. The intermediary handles the conversion logic, preventing direct exposure of system complexity to end users.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The translation system operates autonomously by automatically detecting Pandas operations in user code and converting them to NumPy equivalents without requiring user intervention. The system self-manages the complexity of code translation, optimization strategies, and execution coordination, thereby improving processing speed while minimizing the perceived complexity for users.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS12596549B2Method and system for acceleration of slower data processing codes in machine learning pipelines

Publication Date: 2026.04.07 TATA CONSULTANCY SERVICES LTD

AI summary

Data processing code in machine learning pipelines is primarily done using data frame APIs provided by Pandas and similar libraries. Though, these libraries are easy to use, their temporal performance is worse than similar code written using NumPy or other high-performance libraries. Embodiments herein provide a system and method for acceleration of slower data processing code in machine learning pipelines by automatically generating an accelerated data processing code. Initially, a code is received and pre-processed based on a predefined format to get a standardized code. Further, system identifies code statements having operations that to be performed on a data frame, and an ordered list of data frame columns to generate a filtered dictionary code. Further, a data processing representation is generated using filtering dictionary code and ordered list of data frame columns. Finally, an accelerated data processing code is recommended based on the data processing representation.