Conveyor Element Control Using Local RL for Item Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conveyor systems for piece goods face challenges in efficiently singulating and aligning items with varying properties, as existing control methods are complex and inefficient, especially when adapting to changes in cargo flow or different numbers and lengths of conveyor elements, limiting the ability to maximize throughput and maintain control quality.
Innovation Solution
A computer-implemented method using reinforcement learning to control conveyor elements, where an agent selects actions based on state vectors and action vectors to optimize the speed of each conveyor element, allowing for efficient adaptation to changing cargo conditions and reducing the complexity of high-dimensional control problems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If reinforcement learning is used to control conveyor elements, then adaptability to changing cargo flows is improved, but computational complexity increases
Solution Approach 1:
The control problem is segmented by creating individual state vectors and action vectors for each unit item on the conveyor. Instead of controlling all conveyor elements globally, the system divides control into item-specific segments, where each item receives tailored control actions based on its own state vector, reducing the computational burden of managing high-dimensional control spaces.
Solution Approach 2:
The patent applies local quality by selecting action vectors from an action space that are specific to each unit item's characteristics and position. The control strategy adapts locally to each item's needs rather than applying uniform control, allowing the system to handle diverse cargo properties efficiently while maintaining manageable computational complexity through localized decision-making.
2Adaptability or versatility
If the number of conveyor elements is increased to handle diverse cargo, then versatility is improved, but control complexity increases
Solution Approach 1:
The patent implements universality by using a standardized state vector and action vector structure that can handle any unit item type. The same control framework and vector dimensions are used regardless of cargo diversity, allowing the system to accommodate varying numbers and types of conveyor elements without requiring separate control mechanisms for each scenario.
Solution Approach 2:
The system manages control complexity by changing parameters dynamically through reinforcement learning. The agent learns to adjust control parameters (action vectors) based on observed cargo characteristics and system state, allowing the conveyor system to adapt to diverse cargo types and configurations without increasing the fundamental control structure complexity.
3Manufacturing precision
If manual controller design is used, then control precision is maintained, but adaptability to different cargo properties deteriorates
Solution Approach 1:
The patent applies self-service by implementing a reinforcement learning agent that automatically learns and adapts control strategies for different cargo properties. Instead of requiring manual redesign of controllers for each cargo type, the system self-adjusts by learning from interactions with diverse items, maintaining control precision while gaining adaptability to varying cargo characteristics.
Solution Approach 2:
The control system incorporates feedback mechanisms where the reinforcement learning agent continuously observes cargo behavior and control outcomes, then adjusts its strategy accordingly. This feedback loop enables the system to maintain precision across different cargo types by learning from actual performance data rather than relying on manual tuning for each scenario.
Data Source
Figure 1~2C
Figure 3~4
Figure 5~6
AI summary
The application of reinforcement learning methods to conveyor systems (2) for unit loads (4) quickly reaches its limits due to the large number of individual conveyor elements (12), which determines the dimensionality of the action vectors (a(t)). The invention describes a computer-implemented method, a data processing device, and a computer system for controlling a control unit of a conveyor system (2) with individually controllable conveyor elements (12) to achieve an alignment and/or a defined distance of the unit loads (4), wherein the control of the control unit (14) is determined by an agent acting according to reinforcement learning methods. Based on a representation, an individual, local state vector sn(t) of predefined dimensions, which are identical for all unit loads (4), is created for each of the unit loads (4n).For each individual item (4n), an action vector (an(t)) is selected from an action space according to a strategy (policy) that is the same for all items (4, 4n) for the current state vector (sn(t)) of that item (4). These action vectors (an(t)) are projected onto the conveyor elements (12), whereby conflicts (e.g., multiple action vectors (an(t)) are mapped onto the same conveyor element (12)) are resolved. After a cycle time (Δt), new state vectors (sn(t+Δt)) are created for each item (4n), evaluated with rewards, and the strategy is adjusted.