Distributed Processing Data Format Conversion for ML Libraries

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In conventional distributed processing platforms, using multiple machine learning libraries is cumbersome for users due to the complexity of task definitions and increased burden, as these platforms are not designed to handle different data formats used by various machine learning engines.

Innovation Solution

A distributed processing management apparatus that communicates with multiple execution servers to specify and convert data formats, allowing multiple machine learning libraries to operate on a single platform by converting data to formats compatible with each machine learning engine.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a conventional distributed processing platform is used to execute multiple machine learning libraries, then the platform can process diverse machine learning tasks, but the task definition becomes complicated and the user burden increases

Engineering Contradiction:

Improveability to use multiple machine learning librariesVSAvoidtask definition complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent introduces a data format conversion mechanism as an intermediary layer between the distributed processing platform and multiple machine learning libraries. This converter automatically transforms data between different formats (e.g., LibSVM, CSV, ARFF) and the internal platform format, eliminating the need for users to manually define complex tasks for each library. The conversion process is transparent to users, allowing them to simply specify the desired machine learning library and input data format without dealing with format compatibility issues.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If data format conversion is implemented for each machine learning engine, then multiple machine learning libraries can be supported, but the system complexity increases

Engineering Contradiction:

Improvesupport for multiple machine learning librariesVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements a universal data format conversion mechanism that handles multiple machine learning library formats through a single, unified converter component. This converter is designed to recognize and transform data into the specific formats required by different machine learning libraries (such as LibSVM, CSV, ARFF) using a common conversion framework. By consolidating the conversion functionality into one multi-functional unit rather than separate converters for each library, the system achieves broad format support while minimizing the increase in system complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11656917B2Distributed processing management apparatus, distributed processing method, and computer-readable recording medium

Publication Date: 2023.05.23 NEC CORP

AI summary

A distributed processing management apparatus 10 is connected to a plurality of execution servers 20 so as to be able to communicate with the execution servers 20. The distributed processing management apparatus 10 is provided with a conversion instruction unit 11 configured to specify, for each execution server 20, a data format usable by a machine learning engine executed by the execution server 20, and issue an instruction to convert a data format of data held by the execution server 20 to the specified data format.