Machine Learning Task Construction With Parallel Time-Sliced Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning training processes are inefficient due to sequential data processing, leading to wasted computational resources and slow speeds, as subsequent data cannot be processed until previous data is completed, and single-thread processing is non-interruptible and underutilizes resources.
Innovation Solution
A method and device for constructing and processing machine learning tasks by arranging operation sub-procedures horizontally in a time order with vertical joining, allowing parallel execution of these sub-procedures to utilize computational resources more efficiently, including data processing from multiple sources in sub-intervals and implementing vertical joining relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is processed sequentially in a single-thread manner according to service times, then the processing order is simple and easy to control, but the processing speed is slow and computational resources are underutilized
Solution Approach 1:
The patent divides the data processing task into multiple independent sub-tasks, each handling a specific time interval or data batch. This segmentation allows parallel execution of multiple sub-tasks simultaneously, transforming the sequential single-thread processing into a multi-threaded parallel processing model, thereby significantly improving processing speed while maintaining manageable control through task scheduling.
2Device complexity
If data is processed sequentially in a single-thread manner, then the processing flow is simple, but computational resources are wasted and efficiency is low
Solution Approach 1:
The patent introduces a dynamic task scheduling mechanism that can allocate and reallocate computational resources based on the progress of different sub-tasks. The system dynamically adjusts the execution plan to optimize resource utilization, ensuring that computational resources are actively engaged in processing tasks rather than remaining idle, thus reducing energy waste while managing system complexity through intelligent control.
3Reliability
If subsequent data processing waits for previous data to be completed, then data processing integrity is maintained, but time is wasted and efficiency is reduced
Solution Approach 1:
The patent implements preliminary action by pre-dividing the data into multiple time intervals and preparing corresponding processing sub-tasks in advance. Multiple sub-tasks can be executed in parallel without waiting for previous tasks to complete, as each sub-task processes its designated data portion independently. This maintains data processing integrity through proper data segmentation while eliminating the time waste associated with sequential processing.
4Ease of manufacture
If single-thread processing is used, then the system is simple to implement, but the system cannot be interrupted and resources are underutilized
Solution Approach 1:
The patent segments the single-thread processing system into multiple independent processing threads, each capable of executing specific sub-tasks. This segmentation enables the system to handle interruptions more flexibly, as not all processing must wait for a single thread to complete. The multi-threaded architecture provides adaptability and versatility while maintaining reasonable implementation complexity through standardized thread management mechanisms.
Data Source
AI summary
A method for constructing and processing a machine learning task, a storage medium and an electronic apparatus are provided. The method includes: obtaining sample data configuration information corresponding to the machine learning task; performing arrangement according to the sample data configuration information to obtain a task operation procedure diagram, wherein the task operation procedure diagram includes a plurality of operation sub-procedures, and one operation sub-procedure corresponds to one sub-interval of the target time interval, and is used to obtain sample data in a corresponding sub-interval from the target sample data source and determine machine learning task data based on the sample data; and controlling the plurality of operation sub-procedures to be executed in parallel based on the vertical joining, to obtain target machine learning task data with a vertical joining relationship, and constructing the machine learning task based on the target machine learning task data.