Automated forest harvester control system

Abstract

The disclosure pertains to a forest harvester that includes a harvester control interface and an automation computer. The harvester control interface provides an operator with manual control of the forest harvester. The automation computer is configured to operate the forest harvester according to an action plan. The system includes a control mode with an automatic mode, where actions of the forest harvester are controlled by the automation computer, and amanual mode, where the forest harvester is controlled using the harvester controls. The harvester control interface allows the operator to provide inputs to manually override the automatic mode of the forest harvester. The automation computer updates the action plan based on the operator's inputs.

Description

[0001] Automated Forest Harvester Control System

[0002] Field

[0003] The technology pertains to the field of forestry machinery, specifically focusing on forest harvesters. It also relates to automation systems and control interfaces used in heavy machinery operation. Background

[0004] The field of forest harvesting has seen significant advancements in machinery design and technology over the past century, leading to increased productivity. However, despite these improvements, the technical utilization rate of forest harvesting machines has stagnated at less than 50% over the past two decades. This is not due to a lack of machine capability, but rather the inability of operators to fully utilize the machine's capacity.

[0005] The task of forest harvesting is complex and requires the operator to make hundreds of decisions every minute while simultaneously controlling a large and complex machine. The operator must plan several subtasks in advance, such as assessing the surroundings and terrain, deciding the sequence of trees to cut, determining the placement of the machine, deciding the felling direction for each tree, and deciding where to place different assortments of logs. All these decisions are interrelated and determining the optimal sequence of subtasks is far from trivial.

[0006] This high cognitive load leads to operator fatigue, often within a few hours into a shift, and results in low job satisfaction, high employee turnover, and difficulty in recruiting new operators. Furthermore, new operators require years of on-the-job training, during which productivity is far lower than average. This represents a significant financial risk for small businesses.

[0007] Another issue is the wear and tear on the machines themselves. Production stops due to mechanical failure are costly, and the operator's driving style significantly impacts the longevity and service cost of the machine. While elite operators can generate smooth yet quick motions that utilize most of the machine's capacity without causing excessive strain on mechanical components, most operators struggle with this balance, leading to either reduced production rate or excessive wear.

[0008] These problems contribute to the high cost of wood production and represent significant challenges in the field of forest harvesting.

[0009] Summary According to a first aspect of the disclosure, a forest harvester comprises a harvester control interface with harvester controls that provide an operator with manual control of the forest harvester, an automation computer configured to operate the forest harvester according to an action plan, and a control mode with an automatic mode (where the forest harvester's actions are controlled by the automation computer) and a manual mode (where the forest harvester is controlled using the harvester controls). The harvester control interface allows the operator to manually override the automatic mode, and the automation computer updates the action plan based on the operator's inputs. This setup enables a flexible control system, allowing both automated operation and manual intervention when needed, leading to improved efficiency and adaptability in various situations. Optionally in some examples, the harvester control interface includes a set of joysticks. This offers intuitive and ergonomic manual control of the harvester, enhancing operator comfort and precision.

[0010] Optionally in some examples, the harvester control interface further comprises an assist button that engages or disengages an assistive automation system. This provides a sim- pie way to switch between manual and automated modes, streamlining the workflow and reducing operator workload.

[0011] Optionally in some examples, the assist button is a push-button located on the operator's joystick console. This placement ensures easy access to the assist function without requiring the operator to remove their hands from the primary controls, promoting efficient and uninterrupted operation.

[0012] Optionally in some examples, the forest harvester further comprises a machine actuation interface that receives control signals from the harvester controls and controls the physical movements of the forest harvester based on these signals. This dedicated interface translates operator inputs or automated commands into precise actions, ensuring accurate and responsive control of the harvester's components.

[0013] Optionally in some examples, the forest harvester further comprises interoceptive sen- sors that collect data about the internal state of the forest harvester and provide this data to the automation computer. This feedback loop allows the system to monitor its own performance, adjust operations as needed, and optimize efficiency based on real-time information. Optionally in some examples, the forest harvester further comprises exteroceptive sensors that gather data about the external environment and provide this data to the automation computer. This allows the system to perceive and react to the surrounding conditions, improving safety and enabling automated adjustments for optimal performance in varying terrains and situations. Optionally in some examples, the automation computer comprises a control mode management module that receives signals from a motion planning and control stack and the harvester control interface, prioritizing the latter for controlling the forest harvester. This prioritization ensures that the operator can always take immediate manual control when necessary, overriding automated functions for enhanced safety and flexibility in unex- pected situations.

[0014] Optionally in some examples, the automation computer comprises an intent interpretation module that analyzes the operator's inputs to the harvester control interface to determine the operator's intended operation of the forest harvester. This feature allows the system to understand and anticipate the operator's actions, leading to more seamless collaboration between human and machine and improved overall efficiency.

[0015] Optionally in some examples, the automation computer comprises a motion planning and control stack that receives commands from an action planning function and translates them into control signals for the forest harvester. This module enables the system to execute planned actions precisely and efficiently, optimizing the harvester's movements and operations.

[0016] Optionally in some examples, the motion planning and control stack comprises an action planning function that generates an action plan, providing a sequence of actions for the forest harvester to execute. This structured approach to task execution ensures organized and efficient operation, maximizing productivity and minimizing wasted time or resources. Optionally in some examples, the motion planning and control stack comprises a motion planning and control function that receives the action plan and generates the control signals. This function ensures that the planned actions are translated into precise control inputs for the harvester, enabling accurate and efficient execution of the intended operations. Optionally in some examples, the automation computer comprises an automation software stack that processes sensor data, creates models of the environment and the machine, and manages the planning and control of forest harvester actions. This comprehensive software system integrates all the necessary functions for automated operation, providing a centralized platform for data processing, decision-making, and control execution.

[0017] Optionally in some examples, the forest harvester further comprises a planning interface with an action plan user interface, allowing the operator to adjust the action plan. This feature provides flexibility and control over the automated operations, enabling the operator to modify the plan as needed to adapt to changing conditions or specific requirements. Optionally in some examples, the planning interface further comprises an action plan display that shows the current action plan to the operator. This visual feedback keeps the operator informed about the system's planned actions, promoting situational awareness and allowing for timely intervention if necessary.

[0018] Optionally in some examples, the action plan display is interactive, allowing the operator to adjust the plan directly through the display. This interactive feature simplifies the process of modifying the action plan, providing a user-friendly interface for real-time adjustments and optimized control over the automated operations.

[0019] Optionally in some examples, the forest harvester fells, processes, and sorts tree(s) into designated piles. This automated process streamlines timber harvesting, improving effi- ciency and reducing manual labor.

[0020] Optionally in some examples, the action plan provides a sequence of actions for the forest harvester to execute. This structured approach ensures organized and efficient operation, maximizing productivity and minimizing wasted time or resources.

[0021] Optionally in some examples, the action plan is generated by an action planning function. This automates the planning process, optimizing the sequence of actions for maximum efficiency and adaptability to different situations.

[0022] Optionally in some examples, the action plan is generated in response to the operator providing inputs to the harvester control interface. This allows the operator to influence the automated plan, adapting it to specific needs or preferences for greater control and flexibility.

[0023] Optionally in some examples, the automatic mode is overridden immediately when the operator provides input to the harvester control interface. This prioritizes manual control, ensuring the operator can quickly take over in situations for enhanced safety and respon- siveness.

[0024] According to a second aspect of the disclosure, a method for operating a forest harvester comprises receiving operator inputs at a harvester control interface to control the forest harvester, generating an action plan using an automation computer, controlling actions of the forest harvester using the automation computer in an automatic mode, controlling the forest harvester using the harvester controls in a manual mode, and updating the action plan based on the operator's inputs to the harvester controls. This method enables a flexible and efficient harvesting process, combining automated operations with manual control options for adaptability and responsiveness to various situations. Brief Description of the Drawings

[0025] Examples are described in more detail below with reference to the appended drawings.

[0026] Figure 1 is a system overview showing the assistive automation system, including the automation computer, exteroceptive sensors, planning interface, machine platform, forest harvester, crane, harvester head, interoceptive sensors, machine actuation interface, control signals, and operator.

[0027] Figure 2 is a system architecture showing the assistive automation system, including the automation computer, automation software stack, motion planning and control stack 122, action planning function, action plan, control signals, machine actuation interface, control mode, intent interpretation module, machine model, environment model, perception software, and operator.

[0028] Figure 3 is a forest harvester on the machine platform, showing an action plan example with a tree processing sequence, including crane and harvester head.

[0029] Figure 4 is a sequence of intervention example showing the forest environment with tree, forest harvester, assistive automation system, action plan, and operator. Detailed Description

[0030] The detailed description set forth below provides information and examples of the disclosed technology with sufficient detail to enable those skilled in the art to practice the disclosure. Figure 1 shows a forest harvester 21 in the disclosure figure. The figure illustrates various components of the assistive automation system 10 integrated into the forest harvester 21. The forest harvester 21 is depicted with a crane 22 and a harvester head 23. The machine platform 20 includes the forest harvester 21, crane 22, and harvester head 23. The operator 30 is positioned inside the cabin of the forest harvester 21, interacting with the harvester control interface 25. The interoceptive sensors 24 are also depicted as part of the machine platform 20, providing data about the internal state of the forest harvester 21. The automation computer 12 is illustrated and is responsible for processing data and managing control functions. The exteroceptive sensors 11 are depicted, providing data about the external environment. The planning interface 13 is shown as a separate component, connected to the automation system, providing visual feedback to the operator 30.

[0031] Figure 2 shows a system architecture of the assistive automation system 10. The assistive automation system 10 includes an automation computer 12 and an automation software stack 120. The automation computer 12 comprises a motion planning and control stack 122, an operator 30 intent interpretation module 123, and a control mode management module 124. The motion planning and control stack 122 includes an action planning function 1221 and a motion planning and control function 1222. The action planning function 1221 generates an action plan 1223, which is used by the motion planning and control function 1222 to generate control signals 202. The operator 30 intent interpretation module 123 provides an operator 30 intent estimate 1231 to the action planning function 1221. The control mode management module 124 receives operator inputs 201 and generates control signals 202. The automation software stack 120 includes a perception software 121, which creates an environment model 1011 and a machine model 1012. The en- vironment model 1011 and machine model 1012 are used by the motion planning and control stack 122. The perception software 121 stack receives data from exteroceptive sensors 11 and interoceptive sensors 24. The machine platform 20 includes the interoceptive sensors 24, a harvester control interface 25, and a machine actuation interface 26. The harvester control interface 25 provides operator inputs 201 to the control mode management module 124. The machine actuation interface 26 receives control signals 202 from the control mode management module 124. A planning interface 13 is shown connected to the control mode management module 124, providing visual feedback to the operator 30.

[0032] Figure 3 shows a forest harvester 21 performing a sequence of actions for tree processing. The figure illustrates the forest harvester 21 with a crane 22 and a harvester head 23. The machine platform 20 includes the forest harvester 21, crane 22, and harvester head 23. The operator 30 is positioned inside the cabin of the forest harvester 21. The figure depicts a series of labeled actions (a, b, c, d, el, e2, f, e3, g) corresponding to different stages of the tree processing sequence. The actions are as follows: Move machine actions (a): Move base machine to configuration in which it can reach the next sequence of tree(s) to be cut. Grab tree actions (b): Apply the felling head in a configuration from which the tree can be felled in the planned direction. Felling actions (c): Fell the tree by engaging all necessary motions of the machine (may include motion of felling head, crane 22, and base machine simultaneously). Move felled tree actions (d): Move the tree to the first log assortment pile. Bucking actions(s) (el): Buck first log to the first assortment pile. Bucking actions (e2): Buck second log to the first assortment pile. Move felled tree actions (f): Move the remainder of the tree to the second assortment pile. Buck log (e3): Buck third log to the second assortment pile. Grab tree actions (g): Apply the felling head to the next tree. The action plan 1223 continues over multiple tree processing sequences.

[0033] Figure 4 shows a sequence of intervention. The left side of the figure illustrates a forest harvester 21 with a crane 22 and a harvester head 23, positioned among several trees.

[0034] The operator 30 is depicted inside the cabin of the forest harvester 21. The trees are labeled with different actions points: la, lb, 2a, 2b, 3a, and 3b, indicating various stages of the tree processing sequence. An additional label, a, indicates a manual intervention point where the operator 30 manually moves the harvester head 23 to the left. The as- sistive automation system 10 is also shown as part of the forest harvester 21. The right side of the figure presents a flowchart detailing the sequence of intervention steps. The flowchart is divided into two columns: one for the assistive automation system 10 and one for the operator 30. The assistive automation system 10 column includes the following steps: Automatic Operation under action plan 1223 (a)', 'Disengagement', 'Interpretation of operator 30 intent', 'Recompute action plan 1223 (b)', 'Re-engagement', and 'Automatic Operation under action plan 1223 (b)'. The operator 30 column includes 'Manual intervention (a) (manual movement left)' and 'Re-engagement command'. The flowchart shows the interaction between the operator 30 and the assistive automation system 10 during the intervention sequence. 1 Forest harvester Details

[0035] The forest harvester 21 fells, processes, and sorts tree(s) into designated piles. The forest harvester 21 is operated by an operator 30. The forest harvester 21 may include a machine platform 20, an assistive automation system 10, a crane 22, and a harvester head 23. The crane 22 manipulates the harvester head 23 to perform tasks, such as grabbing and felling tree(s). The harvester head 23 includes mechanisms for grabbing, cutting, and processing tree(s).

[0036] 1.1 Machine platform

[0037] The machine platform 20 includes a harvester control interface 25. In some configurations, the machine platform 20 may also include a machine actuation interface 26 and interocep- tive sensors 24. The machine actuation interface 26 receives control signals 202 from the harvester controls and controls the physical movements of the forest harvester 21 based on these signals. In some instances, the machine actuation interface 26 translates control signals 202 into actuator actions.

[0038] 1.1.1 Harvester control interface The harvester control interface 25 is a component of the machine platform 20. It provides an interface for operator 30 control of the forest harvester 21. The harvester control interface 25 may be easily accessible from the operator's 30 's nominal work position. This may include specifications such as a reach distance from the operator's nominal hand position of less than or equal to 30 cm, an activation force for the assist button between 1 and 5 N, tactile feedback on assist button activation (a perceptible click or vibration), visual confirmation of assist mode activation (a clear indicator on the operator 30 display within 0.5 seconds of activation), and auditory confirmation of assist mode activation (a distinct sound cue within 0.5 seconds of activation). The harvester control interface 25 includes harvester controls. In some instances, the harvester control interface 25 may include an action plan user interface. The action plan user interface may allow the operator 30 to override the existing action plan 1223 of the forest harvester 21. The operator inputs 201 are inputs from the operator 30 to control the forest harvester 21.

[0039] Harvester controls

[0040] The harvester controls provide the operator 30 with manual control of the forest harvester 21 and generate control signals 202 from operator inputs 201. These controls may include joysticks and pedals. In some instances, the harvester controls may also allow the operator 30 to manually override the automated operation of the forest harvester 21. This override functionality prioritizes operator 30 control and ensures that the forest harvester 21 comes under the full control of the operator 30 as soon as they start operating the harvester controls.

[0041] Assist button

[0042] The assist button engages and disengages the assistive automation system 10. The assist button may be a physical push-button located on the operator's joystick console, a toggle switch on the joystick console, a button on a touchscreen interface, a foot pedal, a voice command, a gesture recognized by a camera, a proximity sensor that activates when the operator's hand approaches a designated area, a combination of buttons or switches, or a software-based button on a heads-up display.

[0043] 1.1.2 Interoceptive sensors

[0044] The machine platform 20 may include interoceptive sensors 24. These sensors collect data about the internal state of the forest harvester 21, such as its position, orientation, and forces. The interoceptive sensors 24 provide this data to the automation computer 12. Examples of interoceptive sensors 24 may include IMUs, displacement / angle encoders, force sensors, and pressure / flow sensors. The interoceptive sensors 24 may also include interoceptive sensor data.

[0045] Interoceptive sensor data interoceptive sensor data provides information about the internal state of the forest har- vester 21. This data may include forest harvester 21 position and orientation data, which can be used to calculate the optimal trajectory for moving the harvester head 23 to the next tree. It may also include force sensor data from the harvester head 23, used to detect when the head has a firm grip on a tree. Additionally, it may include hydraulic pressure data for monitoring the stress on the hydraulic system, engine RPM and fuel consumption data for optimizing engine performance, and crane 22 boom angle and extension data for calculating the optimal crane 22 configuration.

[0046] 1.2 Assistive automation system

[0047] The assistive automation system 10 automates forest harvesting operations to reduce operator 30 workload and improve efficiency. It automates control inputs and decision- making processes and plans the future motion of the forest harvester 21. For example, the system may plan optimal trajectories for the harvester head 23 to minimize movement time between tree, addressing the low technical utilization rate caused by the complexity of control interfaces and the lack of automation. It may also plan smooth and efficient crane 22 movements to reduce stress on mechanical components, minimizing wear and tear and addressing the problem of increased wear on machinery due to operator 30 driving styles.

[0048] In some instances, the system may predict potential obstacles or hazardous situations in the environment and plan alternative motions to avoid them , improving safety and reducing downtime caused by accidents or damage. For instance, the assistive automation system 10 may optimize the sequence of actions, such as felling, delimbing, and bucking, to maximize productivity and minimize fuel consumption, addressing the low technical utilization rate caused by operator 30 overload and insufficient automation. The system may also plan motions that account for terrain variations and tree characteristics, ensuring stability and efficient operation even in challenging conditions and mitigating the impact of operator 30 fatigue and inexperience on productivity. In some configurations, the system may dynamically adjust the motion plan based on real-time feedback from sensors, enabling it to adapt to changing conditions and unexpected events, addressing the problem of operator 30 overload by handling unex- pected situations automatically. For example, the system may plan motions that optimize the placement of processed logs into designated piles, improving the efficiency of subsequent forwarding operations and addressing the overall problem of high wood production costs. The assistive automation system 10 includes an automation computer 12. In some instances, the assistive automation system 10 may include exteroceptive sensors 11 and a planning interface 13.

[0049] 1.2.1 Exteroceptive sensors

[0050] The assistive automation system 10 may include exteroceptive sensors 11. These sensors gather data about the external environment of the forest harvester 21, including terrain, tree, and log piles. The exteroceptive sensors 11 provide this data to the automation computer 12. Examples of exteroceptive sensors 11 may include lidar sensors, camera sensors, radar sensors, and GNSS (Global Navigation Satellite System). The exteroceptive sensors 11 may also include exteroceptive sensor data.

[0051] Exteroceptive sensor data exteroceptive sensor data may be real-time, 3D, geo-referenced, high-resolution, and multi-modal (from various sensor types). This data is processed by the perception software 121.

[0052] 1.2.2 Automation computer

[0053] The assistive automation system 10 includes an automation computer 12. The automation computer 12 facilitates seamless switching between manual and automated control. The automation computer 12 may include a control mode management module 124, an intent interpretation module 123, a motion planning and control stack 122, and an automation software stack 120. The automation software stack 120 processes sensor data, creates models of the environment and machine, and manages the planning and control of forest harvester 21 actions. The automation software stack 120 may include perception software 121. The perception software 121 creates an environment model 1011 from exteroceptive sensor data from the exteroceptive sensors 11 and creates a machine model 1012 from data from the interoceptive sensors 24 and optionally from the exteroceptive sensors 11. Control mode management module

[0054] The control mode management module 124 determines if the control mode is automatic or manual and ensures smooth transitions between these modes. It receives signals from the motion planning and control stack 122 and harvester control interface 25. The control mode management module 124 prioritizes the harvester control interface 25 for controlling the forest harvester 21.

[0055] Control mode

[0056] The control mode management module 124 includes a control mode. The control mode governs whether the forest harvester 21 operates under manual or automated control. The control mode may include a manual mode and an automatic mode. In manual mode, the operator 30 has full control of the forest harvester 21. In automatic mode, the automation computer 12 automates the actions and motions of the forest harvester 21.

[0057] Intent interpretation module

[0058] The intent interpretation module 123 analyzes the operator's inputs to the harvester control interface 25 to determine their intended operation of the forest harvester 21. It then adjusts the action plan 1223 based on this interpreted intent. In some instances, the module may use a motion state space distance metric. This module may include an operator intent interpretation algorithm 1231.

[0059] Operator intent interpretation algorithm

[0060] The intent interpretation module 123 may include an operator intent interpretation algo- rithm 1231. The operator intent interpretation algorithm 1231 interprets the operator's 30 's intended actions based on manual control inputs. The algorithm compares the operator's motions to candidate actions. For example, if the operator 30 manually moves the harvester head 23 towards a specific tree, the system analyzes this motion and compares it to pre-defined candidate actions such as 'grab tree A', 'grab tree B', etc. The system then determines the operator's intent by calculating the motion state space distance metric between the operator's motion and the candidate actions. For instance, if the distance metric is smallest for 'grab tree B', the system infers that the operator 30 intends to grab tree B. This eliminates the need for the operator 30 to explicitly select tree B through a complex interface. As another example, if the operator 30 manually adjusts the crane's position and orientation, the system compares this motion with candidate actions such as 'prepare for felling', 'move to next processing location', etc. The system infers the operator's intent based on the similarity between the manual adjustments and the typical crane 22 motions for felling preparation. In some instances, the operator 30 might use manual controls to slightly reposition the harvester head 23 after the system has already initiated a 'grab tree" action(s). The system compares this corrective motion to candidate actions such as 'adjust grip', 'abort grab', etc. Based on the nature of the manual adjustment, the system interprets the operator's intent and incorporates the input into the ongoing action(s). If the operator 30 initiates a manual motion that partially aligns with multiple candidate actions, the system may use the distance metric to determine the most likely intended action(s) or present the operator 30 with a selection interface to confirm the intent. If the operator 30 performs a manual motion that deviates significantly from any pre-defined candidate actions, the system might pause the automated sequence or log the input for later analysis. The operator intent interpretation algorithm 1231 adjusts the action plan 1223 based on the interpreted intent.

[0061] Motion planning and control stack

[0062] The motion planning and control stack 122 receives commands from the action planning function 1221 and translates them into control signals 202 for controlling the forest har- vester 21. It manages the seamless handover between manual and automatic mode and plans future forest harvester 21 motion. This stack may include the action planning function 1221, motion planning and control function 1222, machine model 1012, and environment model 1011. The machine model 1012 provides information about the state and configuration of the forest harvester 21 to the motion planning and control stack 122 and is continuously updated by the perception software 121 stack. The environment model 1011 provides information about the terrain, tree, and log piles to the action planning function 1221 and motion planning and control stack 122 and is continuously updated by the perception software 121 stack.

[0063] Action planning function The motion planning and control stack 122 may include an action planning function 1221. The action planning function 1221 generates an action plan 1223 of actions. It updates the action plan 1223 based on operator 30 input and environment changes.

[0064] Action plan

[0065] The action plan 1223 is the product of the action planning function 1221. The action plan 1223 provides a sequence of actions for the forest harvester 21 to execute. The action plan 1223 may be generated by the action planning function 1221 and may be based on the environment model 1011 and machine model 1012. The action plan 1223 may be a sequence of actions. Actions

[0066] The action plan 1223 includes a series of actions. These actions may include moving the forest harvester 21, moving felled tree to log piles, securing the grapple, grabbing tree, felling tree, and bucking logs.

[0067] Motion planning and control function The motion planning and control stack 122 may include a motion planning and control function 1222. The motion planning and control function 1222 receives the action plan 1223 and generates control signals 202. It uses the machine model 1012 and a motion control algorithm 1225 to plan and execute motions. The motion planning and control function 1222 may include the motion control algorithm 1225 and the control signals 202. The control signals 202 control the forest harvester 21 actuators, such as the crane 22, harvester head 23, and base machine. The control signals 202 may be generated based on the action plan 1223 and machine model 1012.

[0068] Motion control algorithm

[0069] The motion planning and control function 1222 may include a motion control algorithm 1225. The motion control algorithm 1225 plans the forest harvester's 21 's motions. For example, the motion control algorithm 1225 might minimize the time it takes to move the harvester head 23 from one tree to the next, reducing idle time and increasing the number of trees processed per hour. For instance, by optimizing the trajectory and speed of the crane 22 and harvester head 23 movements during felling and processing, the al- gorithm minimizes the time required for each operation. In some instances, the system might simultaneously reposition the base machine while the crane 22 is processing a tree, minimizing the total time for the combined operation. The system may minimize execution time by optimizing the sequence of actions within a task, such as grabbing, felling, and bucking. For example, it might plan the felling direction of a tree to minimize the subsequent movements required for processing and sorting the logs. The motion control algorithm 1225 matches planned and current motion states. It adheres to constraints on wear-related variables and adapts to changes in the environment and operator 30 input. The motion control algorithm 1225 may use continuous replanning. 1.2.3 Planning interface

[0070] The assistive automation system 10 may include a planning interface 13. The planning interface 13 may include an action plan user interface.

[0071] Action plan user interface The action plan user interface allows the operator 30 to adjust the action plan 1223. This interface may include a mouse and / or keyboard, or joysticks, which may be the same joysticks used for the harvester control interface 25. The action plan user interface may also include an action plan display.

[0072] Action plan display An action plan user interface may include an action plan display. The action plan display shows the current action plan 1223 to the operator 30, allowing them to anticipate upcoming forest harvester 21 actions. The action plan display may be interactive, allowing the operator 30 to adjust the action plan 1223. For example, the action plan display may be a touch screen. Other examples of the action plan display may include an AR / VR headset, a head-up display, or a monitor.

[0073] 2 Method For Operating An Assistive Automation System Method Details

[0074] The method for operating the assistive automation system 10 involves a series of steps that seamlessly integrate manual and automated control of the forest harvester 21. The method begins with system initialization, followed by manual operation and system mon- itoring. The operator 30 can then engage the assistive automation mode, allowing the system to execute the action plan 1223. During automated operation, the operator 30 retains the ability to manually override the system if necessary. The system adapts to these manual interventions by interpreting the operator's intent and recomputing the action plan 1223. Finally, the operator 30 can disengage the assistive automation and resume manual control.

[0075] The method prioritizes operator 30 control and safety, allowing for smooth transitions between manual and automated modes. The system continuously monitors the environment and operator 30 actions, adapting to changes and ensuring efficient and safe operation of the forest harvester 21. 2.1 System Initialization

[0076] System initialization is the first step in operating the assistive automation system 10. This process ensures that all components are functioning correctly and ready for operation. The specific steps may vary depending on the configuration of the forest harvester 21 and the assistive automation system 10.

[0077] 2.1.1 Powering On the System

[0078] Powering on the system is the initial step in the system initialization process. This involves activating the power supply to the assistive automation system 10 and the forest harvester 21. The system may perform self-tests and checks to ensure all components are functioning correctly.

[0079] 2.1.2 Sensor Functionality Check

[0080] During the sensor functionality check, the system verifies the operational status of all sensors, both interoceptive and exteroceptive. This check ensures that the sensors are providing accurate and reliable data, which is for the safe and efficient operation of the forest harvester 21.

[0081] The system may perform various tests, such as checking for sensor connectivity, signal integrity, and data validity. For example, the system might check the calibration of the IMUs, the range and accuracy of the lidar sensors, and the clarity and resolution of the camera feeds. The specific tests performed may vary depending on the types of sensors used and the operational requirements of the forest harvester 21.

[0082] If any sensor malfunctions are detected, the system may generate alerts or warnings to notify the operator 30. The system may also have built-in redundancy, allowing it to continue operating with reduced functionality if a sensor fails. In some cases, the system may automatically switch to manual mode if a sensor malfunction is detected, ensuring the safety of the operator 30 and the forest harvester 21.

[0083] 2.1.3 Ensuring System Readiness

[0084] Ensuring system readiness involves verifying the functionality of all hardware and software components of the assistive automation system 10. This step follows the sensor functionality check and precedes manual operation and system monitoring. The system checks the communication links between the automation computer 12, sensors, and actuators. It also verifies the integrity of the software, including the operating system, control algorithms, and user interface. The system may run diagnostics to identify any potential issues and ensure that all components are working as expected.

[0085] Once the system readiness check is complete, the system is ready for manual operation.

[0086] The operator 30 can then begin controlling the forest harvester 21 using the harvester control interface 25.

[0087] 2.2 Manual Operation and System Monitoring

[0088] During manual operation, the operator 30 has full control of the forest harvester 21 using the harvester control interface 25. The system continuously monitors the operator's actions) and the environment, collecting data that can be used to optimize the action plan 1223 and improve the performance of the assistive automation system 10.

[0089] 2.2.1 Operator Control of the Forest Harvester

[0090] In manual operation, the operator 30 has complete control over the forest harvester 21 using the harvester control interface 25. This interface typically includes joysticks, pedals, and other controls that allow the operator 30 to manipulate the crane 22, harvester head 23, and base machine.

[0091] The operator 30 uses the harvester control(s) to perform various tasks, such as moving the forest harvester 21, selecting tree for felling, and controlling the cutting and processing operations. The system monitors the operator's action(s) and the environment, collecting data that can be used to optimize the action plan 1223 and improve the performance of the assistive automation system 10.

[0092] Manual operation allows the operator 30 to handle unexpected situations or tasks that are not yet automated. The operator 30 can switch to assistive automation mode at any time by activating the assist button.

[0093] 2.2.2 System Monitoring of Operator Actions and Environment During both manual and automated operation, the system continuously monitors the operator's action(s) and the surrounding environment. This monitoring process collects data that is used to improve the performance and safety of the assistive automation system 10.

[0094] The system monitors the operator's inputs to the harvester control interface 25, such as joystick movements and button presses. It also monitors the state of the forest harvester 21, including the position and orientation of the crane 22, harvester head 23, and base machine. Additionally, the system monitors the environment using exteroceptive sensors 11, collecting data about the terrain, tree(s), and log piles.

[0095] The collected data is used by the intent interpretation module 123 to understand the operator's intentions and adjust the action plan 1223 accordingly. The data is also used by the motion planning and control stack 122 to optimize the movements of the forest harvester 21 and ensure safe and efficient operation.

[0096] 2.3 Engaging Assistive Automation Mode

[0097] The operator 30 engages the assistive automation mode by activating the assist button.

[0098] The system confirms the engagement and begins executing the current action plan 1223. The system continues to monitor the environment and provides feedback to the operator 30.

[0099] 2.3.1 Activation of Assist Button

[0100] The operator 30 activates the assist button to engage the assistive automation mode. The assist button is located on the harvester control interface 25, typically within easy reach of the operator's hand.

[0101] The assist button may be a physical push-button, a toggle switch, or a software-based button on a touchscreen interface. The specific design of the assist button may vary depending on the configuration of the harvester control interface 25.

[0102] Activating the assist button sends a signal to the control mode management module 124, which then initiates the transition to assistive automation mode.

[0103] 2.3.2 System Confirmation of Assist Mode Engagement

[0104] Upon receiving the signal from the assist button, the system confirms the engagement of assistive automation mode. This confirmation may involve visual and auditory cues to inform the operator 30 that the system is now in automated control. The system may display a message on the action plan display or provide an audible alert. The specific confirmation method may vary depending on the configuration of the system .

[0105] Once the assist mode is confirmed, the system begins executing the current action plan 1223. 2.3.3 Execution of Current Action Plan

[0106] After confirming the engagement of assistive automation mode, the system begins executing the current action plan 1223. The action plan 1223 consists of a sequence of actions for the forest harvester 21 to perform, such as moving to a specific location, grabbing a tree, felling it, and processing the logs.

[0107] The motion planning and control stack 122 translates the actions in the action plan 1223 into control signals 202 for the actuators of the forest harvester 21. The system continuously monitors the environment and the state of the forest harvester 21 to ensure that the actions are executed safely and efficiently. The operator 30 can manually override the automated operation at any time by using the harvester control interface 25.

[0108] 2.3.4 Continuous Monitoring and Feedback Provision

[0109] Throughout the operation of the assistive automation system 10, the system continuously monitors the environment and the state of the forest harvester 21. This monitoring process collects data that is used to provide feedback to the operator 30 and to adapt the action plan 1223 as needed.

[0110] The system may provide visual feedback to the operator 30 through the action plan display, showing the current status of the action plan 1223 and any potential issues. The system may also provide auditory feedback through alerts or warnings. The collected data is used by the intent interpretation module 123 and the motion planning and control stack 122 to adjust the action plan 1223 and optimize the movements of the forest harvester 21. This continuous monitoring and feedback loop ensures that the system operates safely and efficiently, even in changing environmental conditions.

[0111] 2.4 Automated Operation with Manual Override In automated operation, the system executes the planned actions based on the action plan 1223. The operator 30 can manually override the automated operation at any time by using the harvester control interface 25. This override functionality prioritizes operator 30 control and allows for immediate intervention in unexpected situations. 2.4.1 Execution of Planned Actions

[0112] 2.4.2 Manual Override of Automated Operation

[0113] 2.5 Disengaging Assistive Automation and Resuming Manual Control

[0114] The operator 30 can disengage the assistive automation at any time and resume manual control of the forest harvester 21. This can be done by deactivating the assist button or through another designated input. The system seamlessly transitions back to manual mode, ensuring a smooth handover of control.

[0115] 2.5.1 Manual Disengagement of Assistive Automation

[0116] 2.5.2 Reversion to Manual Control 2.6 System Adaptation and Re-engagement of Assistive Automation

[0117] When the operator 30 manually intervenes during automated operation, the system adapts by interpreting the operator's intent and recomputing the action plan 1223. This adaptation allows the system to learn from the operator's actions and improve its performance over time. The operator 30 can then re-engage the assistive automation mode, and the system will continue operation based on the updated action plan 1223.

[0118] 2.6.1 Interpretation of Operator Intent

[0119] 2.6.2 Recomputation of Action Plan

[0120] 2.6.3 Re-engagement of Assistive Automation Mode

[0121] 3 Operational Process The operational process of the forest harvester 21 combines automated actions with manual control by the operator 30. The system initializes by powering on and checking sensor functionality, ensuring all components are ready. The operator 30 then assumes manual control, performing tasks such as moving the harvester and selecting tree(s). During this phase, the system monitors the operator's actions and the environment. The operator 30 can engage the assistive automation mode, which executes a preplanned sequence of actions, the action plan 1223. This plan considers factors like terrain, tree characteristics, and log placement. The system continuously monitors the environment and provides feedback to the operator 30.

[0122] Even in automated mode, the operator 30 retains full control and can manually override the system at any time. This override is prioritized, allowing immediate intervention if needed. The system responds to manual interventions by interpreting the operator's intent and adjusting the action plan 1223 accordingly. The operator 30 can then re-engage the assistive automation mode, and the system continues operation based on the updated plan. This iterative process allows for efficient harvesting while maintaining operator 30 control and safety.

[0123] 3.1 Forest Harvester Movement and Positioning

[0124] The forest harvester's 21 's movement and positioning are for efficient tree processing. In manual mode, the operator 30 controls the harvester's base machine, crane 22, and har- vester head 23 using the harvester control interface 25. This interface includes joysticks, pedals, and other controls, allowing precise maneuvering within the forest environment.

[0125] In automated mode, the motion planning and control stack 122 determines the harvester's movements based on the action plan 1223. This plan considers factors such as the location of tree(s), terrain, and designated log piles. The system calculates optimal trajectories and generates control signals 202 for the harvester's actuators, ensuring efficient and safe movement.

[0126] The interoceptive sensors 24 provide data on the harvester's internal state, including position, orientation, and forces. This data is used by the motion control algorithm 1225 to continuously adjust movements and maintain stability, especially in challenging terrain. The exteroceptive sensors 11 gather data about the external environment, such as terrain variations and obstacles, which is integrated into the motion planning process.

[0127] The system prioritizes smooth and efficient movements to minimize wear and tear on the machinery. It also accounts for factors like crane 22 reach and harvester head 23 positioning to optimize tree processing operations.

[0128] 3.2 Tree Processing Sequence

[0129] The tree processing sequence involves a series of automated actions executed by the forest harvester 21. The sequence typically begins with the harvester moving to a suitable position near the target tree. The crane 22 then maneuvers the harvester head 23 to grab the tree securely. Once the tree is gripped, the felling action is initiated, involving a combination of harvester head 23 and crane 22 movements. After felling, the tree is moved to a designated processing area. Here, the delimbing and bucking actions take place, separating the trunk into logs of desired lengths. These logs are then sorted and placed into designated piles based on their size and quality.

[0130] The system optimizes the entire sequence to maximize efficiency and minimize fuel consumption. It considers factors such as tree size, felling direction, and log assortment requirements. The action plan 1223 guides the sequence, ensuring smooth transitions between actions and minimizing idle time. The system continuously monitors the process, adjusting parameters as needed based on real-time feedback from sensors.

[0131] 3.3 Manual Intervention and System Response

[0132] The operator 30 can manually intervene in the automated tree processing sequence at any time. This intervention might involve adjusting the harvester's position, modifying the crane 22 or harvester head 23 movements, or changing the log sorting criteria.

[0133] When the operator 30 provides manual input, the system immediately disengages the automated mode and prioritizes the operator's commands. The intent interpretation module 123 analyzes the operator's actions to understand their intent. This module uses a mo- tion state space distance metric to compare the operator's movements with predefined candidate actions.

[0134] Based on the interpreted intent, the system adjusts the action plan 1223. For example, if the operator 30 manually moves the harvester head 23 towards a specific tree, the system might infer that the operator 30 intends to process that tree next. The action plan 1223 is updated to reflect this change.

[0135] The system provides feedback to the operator 30 throughout the intervention process, confirming the interpreted intent and displaying the updated action plan 1223. This feedback loop ensures that the system and the operator 30 are in sync, promoting efficient and safe operation. 3.4 Action Plan Adjustment and Re-engagement

[0136] Following a manual intervention, the action plan 1223 is adjusted based on the operator's 30 's input. The intent interpretation module 123 analyzes the operator's actions and updates the plan accordingly. This adjustment might involve adding new actions, removing existing actions, or changing the order of actions. The system then presents the updated action plan 1223 to the operator 30 through the action plan display. The operator 30 can review the changes and make further adjustments if needed. The action plan display is interactive, allowing the operator 30 to modify the plan using the harvester control interface 25 or a dedicated planning interface 13.

[0137] Once the operator 30 is satisfied with the updated plan, they can re-engage the assistive automation mode. The system then resumes automated operation, executing the adjusted action plan 1223. This process allows for seamless integration of manual input 5 and automated control, ensuring efficient and flexible operation of the forest harvester 21.

[0138] Item 1. A forest harvester 21 comprising: a harvester control interface 25 comprising: harvester controls, wherein the harvester controls provide an operator 30 with manual control of the forest harvester 21;

[0139] 0 an automation computer 12 configured to operate the forest harvester 21 according to an action plan 1223, a control mode comprising: an automatic mode, wherein the actions of the forest harvester 21 are controlled by the automation computer 12;

[0140] 5 a manual mode, wherein the forest harvester 21 are controlled using the harvester controls; and wherein the harvester control interface 25 allows the operator 30 to provide inputs to manually override the automatic mode of forest harvester 21, and wherein the automation computer 12 updates the action plan 1223 in depones dence on the operator 30 inputs.

[0141] Item 2. The forest harvester 21 according to item 1, wherein the harvester control interface 25 includes a set of joysticks.

[0142] Item 3. The forest harvester 21 according to item 1 or 2, wherein the harvester control interface 25 further comprises an assist button, wherein the assist button en- 5 gages / disengages an assistive automation system 10.

[0143] Item 4. The forest harvester 21 according to item 3, wherein the assist button is a pushbutton located on the operator's joystick console.

[0144] Item 5. The forest harvester 21 according to any of item 1 to 4, wherein the forest harvester 21 further comprises a machine actuation interface 26, wherein the macs chine actuation interface 26 receives control signals 202 from the harvester controls and controls physical movements of the forest harvester 21 in dependence on the control signals 202.

[0145] Item 6. The forest harvester 21 according to any of items 1 to 5, wherein the forest har- vester 21 further comprises a set of interoceptive sensors 24, wherein the interoceptive sensors 24 collect interoceptive sensor data about an internal state of the forest harvester 21 and provide the interoceptive sensor data to the automation computer 12. Item 7. The forest harvester 21 according to any of items 1 to 6, wherein the forest harvester 21 further comprises a set of exteroceptive sensors 11, wherein the exteroceptive sensors 11 gather exteroceptive sensor data about an external environment of the forest harvester 21 and provide the exteroceptive sensor data to the automation computer 12. Item 8. The forest harvester 21 according to any of items 1 to 7, wherein the automation computer 12 comprises a control mode management module 124, wherein the control mode management module 124 receives signals from a motion planning and control stack 122 and the harvester control interface 25 and prioritizes the harvester control interface 25 for controlling the forest harvester 21. Item 9. The forest harvester 21 according to any of items 1 to 8, wherein the automation computer 12 comprises an intent interpretation module 123, wherein the intent interpretation module 123 analyzes operator's 30's inputs to the harvester control interface 25 to determine an operator's 30's intended operation of the forest harvester 21.

[0146] Item 10. The forest harvester 21 according to any of items 1 to 9, wherein the automation computer 12 comprises a motion planning and control stack 122, wherein the motion planning and control stack 122 receives commands from an action planning function 1221 and translates the commands into control signals 202 for controlling the forest harvester 21. Item 11. The forest harvester 21 according to item 10, wherein the motion planning and control stack 122 comprises an action planning function 1221, wherein the action planning function 1221 generates an action plan 1223 of actions, wherein the action plan 1223 provides a sequence of actions for the forest harvester 21 to execute. Item 12. The forest harvester 21 according to any of items 10 to 11, wherein the motion planning and control stack 122 comprises a motion planning and control function 1222, wherein the motion planning and control function 1222 receives the action plan 1223 and generates the control signals 202. Item 13. The forest harvester 21 according to any of items 1 to 12, wherein the automation computer 12 comprises an automation software stack 120, wherein the automation software stack 120 processes sensor data, creates models of an environment and a machine, and manages the planning and control of forest harvester 21 actions.

[0147] Item 14. The forest harvester 21 according to any of items 1 to 13, further comprising a planning interface 13 comprising an action plan user interface, wherein the action plan user interface allows an operator 30 to adjust the action plan 1223.

[0148] Item 15. The forest harvester 21 according to item 14, wherein the planning interface 13 further comprises an action plan display, wherein the action plan display displays a current action plan 1223 to the operator 30.

[0149] Item 16. The forest harvester 21 according to item 15, wherein the action plan display is interactive and configured to allow the operator 30 to adjust the plan using the action plan display. Item 17. The forest harvester 21 according to any of items 1 to 16, wherein the forest harvester 21 fells, processes, and sorts tree(s) into designated piles.

[0150] Item 18. The forest harvester 21 according to any of items 1 to 17, wherein the action plan 1223 provides a sequence of actions for the forest harvester 21 to execute.

[0151] Item 19. The forest harvester 21 according to item 18, wherein the action plan 1223 is generated by an action planning function 1221.

[0152] Item 20. The forest harvester 21 according to item 18 or 19, wherein the action plan 1223 is generated in response to the operator 30 providing inputs to harvester control interface 25.

[0153] Item 21. The forest harvester 21 according to any of items 1 to 20, wherein the automatic mode is overridden immediately when the operator 30 provides input to the harvester control interface 25.

[0154] Item 22. A method for operating a forest harvester 21 comprising: receiving operator 30 inputs at a harvester control interface 25 to control the forest harvester 21; generating an action plan 1223 using an automation computer 12; controlling actions of the forest harvester 21 using the automation computer 12 in an automatic mode; controlling the forest harvester 21 using the harvester controls in a manual mode; and updating the action plan 1223 in dependence on the operator 30 inputs to the harvester controls. The terminology used herein is for the purpose of describing particular aspects only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. It will be further understood that the terms "comprises," "comprising," "includes," and / or "including" when used herein specify the presence of stated features, integers, actions, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, actions, steps, operations, elements, components, and / or groups thereof.

[0155] It will be understood that, although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the scope of the present disclosure.

[0156] Relative terms such as "below" or "above" or "upper" or "lower" or "horizontal" or "vertical" may be used herein to describe a relationship of one element to another element as illustrated in the Figures. It will be understood that these terms and those discussed above are intended to encompass different orientations of the device in addition to the orientation depicted in the Figures. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or intervening elements may be present. In contrast, when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements present.

[0157] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0158] It is to be understood that the present disclosure is not limited to the aspects described above and illustrated in the drawings; rather, the skilled person will recognize that many changes and modifications may be made within the scope of the present disclosure and appended claims. In the drawings and specification, there have been disclosed aspects for purposes of illustration only and not for purposes of limitation, the scope of the disclosure being set forth in the following claims.

Claims

Claims1. A forest harvester (21) comprising: a harvester control interface (25) comprising: harvester controls, wherein the harvester controls provide an operator (30) with manual control of the forest harvester (21); an automation computer (12) configured to operate the forest harvester (21) according to an action plan (1223), a control mode comprising: an automatic mode, wherein the actions of the forest harvester (21) are controlled by the automation computer (12); a manual mode, wherein the forest harvester (21) is controlled using the harvester controls; and wherein the harvester control interface (25) allows the operator (30) to provide inputs to manually override the automatic mode of forest harvester (21), and wherein the automation computer (12) updates the action plan (1223) in dependence on the operator (30) inputs.

2. The forest harvester (21) according to claim 1, wherein the harvester control interface (25) includes a set of joysticks.

3. The forest harvester (21) according to claim 1 or 2, wherein the harvester control interface (25) further comprises an assist button, wherein the assist button engages / dis- engages an assistive automation system (10).

4. The forest harvester (21) according to claim 3, wherein the assist button is a pushbutton located on the operator's joystick console.

5. The forest harvester (21) according to any of claim 1 to 4, wherein the forest harvester (21) further comprises a machine actuation interface (26), wherein the machine actuation interface (26) receives control signals (202) from the harvester controls and controls physical movements of the forest harvester (21) in dependence on the control signals (202).

6. The forest harvester (21) according to any of claims 1 to 5, wherein the forest harvester (21) further comprises a set of interoceptive sensors (24), wherein the interoceptive sensors (24) collect interoceptive sensor data about an internal state of the forest harvester (21) and provide the interoceptive sensor data to the automation computer (12).

7. The forest harvester (21) according to any of claims 1 to 6, wherein the forest harvester (21) further comprises a set of exteroceptive sensors (11), wherein the exteroceptive sensors (11) gather exteroceptive sensor data about an external environment of the forest harvester (21) and provide the exteroceptive sensor data to the automation computer (12).

8. The forest harvester (21) according to any of claims 1 to 7, wherein the automation computer (12) comprises a control mode management module (124), wherein the control mode management module (124) receives signals from a motion planning and control stack (122) and the harvester control interface (25) and prioritizes the harvester control interface (25) for controlling the forest harvester (21).

9. The forest harvester (21) according to any of claims 1 to 8, wherein the automation computer (12) comprises an intent interpretation module (123), wherein the intent interpretation module (123) analyzes operator's (30)'s inputs to the harvester control interface (25) to determine an operator's (30)'s intended operation of the forest harvester (21).

10. A method for operating a forest harvester (21) comprising: receiving operator (30) inputs at a harvester control interface (25) to control the forest harvester (21); generating an action plan (1223) using an automation computer (12); controlling actions of the forest harvester (21) using the automation computer (12) in an automatic mode; controlling the forest harvester (21) using the harvester controls in a manual mode; and updating the action plan (1223) in dependence on the operator (30) inputs to the harvester controls.

Images