Control of a rock bolting process performed by a mining and / or construction machine

Abstract

A mining and / or construction machine, a control device, and a method implemented therein for controlling operation of the machine for drilling and rock bolting a plurality of boreholes in a rock surface, the machine comprising at least one boom, a drilling and bolting implement coupled to the at least one boom, a detection system comprising at least one movable imaging device, and a fluid injection arrangement. The method comprises obtaining first imaging data acquired by the movable imaging device from a first portion of the rock surface including a planned borehole location; identifying within the first portion, based at least on the planned borehole location and the first imaging data, a focus region encompassing a position of a distal end of the drilling and bolting implement at the planned borehole location; controlling the imaging device to acquire second imaging data from the focus region; and controlling the fluid injection arrangement in response to a determined status of injection of the fluid into the one borehole by the fluid injection arrangement.

Description

Docket No.: DRS2993PC00TITLE CONTROL OF A ROCK BOLTING PROCESS PERFORMED BY A MINING AND / OR CONSTRUCTION MACHINETECHNICAL FIELD

[0001] The disclosure relates to control of a mining and / or construction machine, more particularly to a method and a control device for controlling at least one imaging device of the mining and / or construction machine to monitor insertion of rock bolts into boreholes drilled in a rock surface, and to monitor and control a fluid injection arrangement configured to inject a binding fluid into the boreholes to secure the rock bolts therein. The disclosure also relates to the mining and / or construction machine comprising the at least one imaging device, a drilling and bolting arrangement, the fluid injection arrangement and the control device.BACKGROUND

[0002] Rock bolting is a widely used method for reinforcing and stabilizing rock mass in mining, civil engineering, and tunneling applications. It involves inserting steel bolts into drilled holes or boreholes in rock surfaces to provide support and prevent rockfall or collapse. Recent advances have focused on mining and / or construction machines for automated rock bolting, which are capable of autonomously or remotely controlled drilling of a borehole and anchoring a bolt in the borehole. Conventional rock bolts may be steel rods that are anchored into drilled holes by injection of a binding fluid or anchoring agent, such as a resin-based grout or any other binding material. The bolts work by transferring load from the unstable rock mass to the more stable surrounding material, thus preventing movement or collapse of the rock strata.

[0003] In mining, tunnelling, and construction, developments are underway towards performing operations that are autonomously or remotely controlled. For example, technologies are being developed to perform drilling and / or rock bolting without involvement by an operator on a mining and / or construction machine. However, performing autonomous or remotely controlled operations requires accurate monitoring of the operations, which may be a challenging task, particularly given specifics of a mining or similar environment.

[0004] In rock bolting, further challenges occur due to injection of a binding fluid into in a borehole, since a volume of the borehole may be unknown. Conventional approaches to rock bolting may include visual assessment by a human operator of a progress of the injection process or injecting a predefined volume of the binding fluid into a borehole, which may lead to securing a bolt in the borehole in an inadequate manner when the amount of the injected fluid is not sufficient. As another potential problem, leakages of the binding fluid and thus its waste may occur, which may not be recognized in automated rock bolting.

[0005] Accordingly, there is a need for improved methods, devices and mining and / or construction machines for rock bolting. In particular, there is a need for control techniques for operating a mining and / or construction machine in a manner that allows monitoring rock drilling and bolting, as well as fluid injection, with accuracy appropriate for autonomous and / or remotely controlled operations.Docket No.: DRS2993PC00SUMMARY

[0006] Accordingly, the object of the present disclosure is to provide techniques for controlling operation of a mining and / or construction machine for drilling and rock bolting a plurality of boreholes in a rock surface, in the manner that allows accurate and reliable monitoring of the drilling and rock bolting operation and of a process of injection of a binding fluid into a borehole.

[0007] In an aspect, a method for controlling operation of a mining and / or construction machine for drilling and rock bolting a plurality of boreholes in a rock surface is provided. The mining and / or construction machine comprises at least one boom, a drilling and bolting implement coupled to the at least one boom, a detection system comprising at least one movable imaging device, and a fluid injection arrangement for injecting a fluid into the plurality of boreholes. The method comprises obtaining first imaging data acquired by the at least one movable imaging device from a first portion of the rock surface including a planned borehole location, wherein one borehole is to be drilled approximately at the planned borehole location; identifying, within the first portion of the rock surface, based at least on the planned borehole location and on the first imaging data, a focus region encompassing a position of a distal end of the drilling and bolting implement at the planned borehole location; controlling the at least one movable imaging device to acquire second imaging data from the focus region; analyzing the second imaging data to determine a status of injection of the fluid into the one borehole, which injection is performed by the fluid injection arrangement; and controlling the fluid injection arrangement in response to the determined status of injection of the fluid.

[0008] Technical benefits may include reliable, automatic assessment of the fluid injection status. The technical benefit may include improved control of the fluid injection process and detection of irregularities in the injection process, such as leakages. These benefits may be particularly important when the exact volume of the borehole is unknown or when the surrounding material is porous. The automatic assessment of the status of injection of the fluid into the one borehole facilitates an automatic and / or autonomous rock bolting, by accurately and reliably ensuring a rock bolt or anchor inserted in the borehole is surrounded by a sufficient amount of binding fluid and, as another advantage, a risk of wasting the binding fluid is reduced or eliminated. Furthermore, a need for manual control of operation of the fluid injection arrangement is reduced or eliminated, which improves the entire rock bolting process and thus improves efficiency and productivity of mining operations.

[0009] In some examples, the method may comprise analyzing the second imaging data to determine whether the focus region comprises a surface element in the focus region, wherein the surface element is excluded from consideration when analyzing the second imaging data to determine a status of injection of the fluid into the one borehole.

[0010] In some examples, the surface element may be at least one selected from the group consisting of a previously installed bolt part, a residue from a previous fluid injection, and a geological feature in the focus region. Other surface elements may be detected and recognized in the focus region.

[0011] In some examples, the method may comprise initiating the fluid injection arrangement to begin injecting the fluid into the one borehole after the second imaging data is analyzed to determine whether the focusDocket No.: DRS2993PC00region comprises a surface element in the focus region. The surface element is excluded from consideration when analyzing the second imaging data to determine the status of injection of the fluid into the one borehole.

[0012] Technical benefits may include discerning whether there are elements in the focus region that should not be considered during analysis of the second imaging data, since they are different from the fluid that is being currently injected into the borehole. In this way, accuracy, reliability, and speed of the detection of the fluid may be increased, whereby control of the fluid injection process is improved.

[0013] In some examples, analyzing the second imaging data to determine the status of injection of the fluid into the one borehole may comprise applying a machine-learning model to the second imaging data.

[0014] In some examples, the machine-learning model may comprise at least one deep learning model.

[0015] In some examples, the at least one deep learning model may comprise a convolutional neural network (CNN) model. In some examples, the at least one deep learning model may comprise a semantic segmentation network model.

[0016] Technical benefits of applying the machine-learning model, such as e.g. a deep learning model, to the second imaging data to determine the status of injection of the fluid into the one borehole may include greatly improved accuracy and performance, versatility, robustness to noise, and scalability of the assessment of the second imaging data. A deep learning model can outperform traditional vision algorithms in recognizing when a binding fluid, that is currently being injected into the borehole, appears outside of the borehole, at a certain location.

[0017] In some examples, the method may comprise determining a position and angle of one or more joints of the at least one boom having the drilling and bolting implement coupled thereto, and identifying the focus region further based on the position and angle of one or more joints of the boom.

[0018] In some examples, the method may comprise, as the second imaging data is being analyzed to determine the status of injection of the fluid into the one borehole, dynamically adjusting the focus region based at least on one or more characteristics of an environment in the focus region.

[0019] Technical benefits may include assessing an area having a wider boundary than in case when a position, size and boundary if the focus region are fixed. Thus, the binding fluid can advantageously be detected at various positions which may be offset from a distal end of the fluid injection arrangement and from a collar of the borehole.

[0020] In some examples, determining the status of injection of the fluid may comprise detecting fluid outside of the one borehole, the method comprising, in response to the detection of the fluid outside of the one borehole, controlling the fluid injection arrangement to terminate the injection of the fluid into the one borehole. In such examples, a successful filling of the borehole may be automatically detected.

[0021] In some examples, determining the status of injection of the fluid may comprise detecting a leakage in the fluid injection arrangement, the method comprising, in response to the detection of the leakage, controlling the fluid injection arrangement to terminate the injection of fluid into the one borehole and generate an error signal. In such examples, an error signal or information indicative of an error, e.g., in operation of the fluidDocket No.: DRS2993PC00injection arrangement may be automatically detected. This may indicate that a filling of the borehole is not completed.

[0022] Technical benefits may include discerning different scenarios related to appearance of the binding fluid outside of the borehole, so that the mining and / or construction machine can be controlled to take appropriate actions. In some examples, a position of the detected fluid may indicate that there is a leakage from the fluid injection arrangement, whereas an expected position of the detected fluid at a collar of the borehole may be indicative of a successful filling of the borehole. In some examples, a position of the detected fluid may vary in cases when a successful filling of the borehole occurs, and the method may be configured to automatically recognize such scenarios and generate a corresponding information, control instruction or perform other actions.

[0023] In some examples, controlling the at least one movable imaging device to acquire the second imaging data from the focus region may be performed in response to initiating the fluid injection arrangement to begin injecting the fluid into the one borehole.

[0024] In some examples, the method may comprise selecting an imaging device of the at least one movable imaging device to acquire the second imaging data from the focus region. The imaging device may comprise one or more imaging devices.

[0025] Technical benefits may include determining, automatically in some cases, which one or more of the at least one movable imaging device can be controlled to acquire the second imaging data with a good performance. The imaging devices may be positioned at different locations of the mining and / or construction machine so that the imaging devices may acquire imaging data from different angles and at different positions. The method may determine which of one or more of the imaging devices are most suitable for monitoring the focus region.

[0026] In an aspect, a control device for controlling operation of a mining and / or construction machine for drilling and rock bolting a plurality of boreholes in a rock surface is provided. The mining and / or construction machine comprises at least one boom, a drilling and bolting implement coupled to the at least one boom, a detection system comprising at least one movable imaging device, and a fluid injection arrangement for injecting a fluid into the plurality of boreholes. The control device is configured to perform the method in accordance with any aspects and examples of the present disclosure.

[0027] The technical benefits of the control device may be the same or similar to those achieved by the method in accordance with any one or more examples of the present disclosure, as discussed above. Further, all embodiments of the method are applicable to and combinable with all embodiments of the control device according to the examples herein, and vice versa.

[0028] In an aspect, a mining and / or construction machine for drilling and rock bolting a plurality of boreholes in a rock surface is provided. The mining and / or construction machine comprises at least one boom, a drilling and bolting implement coupled to the at least one boom, a detection system comprising at least one movable imaging device, a fluid injection arrangement for injecting a fluid into the plurality of boreholes, and a control device for controlling operation of the mining and / or construction machine. The control device is configured to perform the method in accordance with any aspects and examples of the present disclosure.Docket No.: DRS2993PC00

[0029] The technical benefits of the mining and / or construction machine may be the same or similar to those achieved by the method in accordance with any one or more examples of the present disclosure, as discussed above. Further, all embodiments of the method are applicable to and combinable with all embodiments of the mining and / or construction machine according to the examples herein, and vice versa.

[0030] In an aspect, a computer-program product comprising computer-executable instructions is provided. The computer-executable instructions, when executed by at least one processor, cause the at least one processor to perform the method in accordance with any of the examples of the present disclosure.

[0031] The technical benefits of the computer-program product may be the same or similar to those achieved by the method in accordance with any one or more examples of the present disclosure, as discussed above. Further, all embodiments of the method are applicable to and combinable with all embodiments of the computer-program product according to the examples herein, and vice versa.

[0032] In an aspect, a non-transitory tangible computer-readable medium is provided that comprises computer-executable instructions that, when executed by at least one processor, cause the at least one processor to perform the method in accordance with any of the examples of the present disclosure.

[0033] The technical benefits of the computer-readable medium may be the same or similar to those achieved by the method in accordance with any one or more examples of the present disclosure, as discussed above. Further, all embodiments of the method are applicable to and combinable with all embodiments of the computer-readable medium according to the examples herein, and vice versa.

[0034] Additional features and advantages are disclosed in the following description, claims, and drawings. Furthermore, additional advantages will be readily apparent from the present disclosure to those skilled in the art or recognized by practicing the disclosure as described herein. There are also disclosed herein control units, computer program products, and computer-readable media associated with the above discussed technical effects and corresponding advantages.BRIEF DESCRIPTION OF THE DRAWINGS

[0035] With reference to the appended drawings, below follows a more detailed description of aspects of the disclosure cited as examples.

[0036] FIG. 1 is a schematic diagram illustrating a mining and / or construction machine, in accordance with an example of the present disclosure.

[0037] FIG. 2 is a schematic diagram illustrating a fluid injection arrangement in accordance with an example of the present disclosure.

[0038] FIGs. 3A, 3B, and 3C are schematic diagrams illustrating the fluid injection arrangement of FIG. 1 when used an automated rock bolting process in accordance with an example of the present disclosure.

[0039] FIG. 4 is a flowchart illustrating a method for controlling operation of a mining and / or construction machine for drilling and rock bolting a plurality of boreholes in a rock surface, in accordance with an example of the present disclosure.Docket No.: DRS2993PC00

[0040] FIG. 5 is a flowchart illustrating a method for training a deep learning model for determining a status of injection of a fluid into a borehole, which injection is performed by the fluid injection arrangement, in accordance with an example of the present disclosure.

[0041] FIGs. 6A and 6B are schematic block diagrams illustrating an example of a computer system in which a method in accordance with examples of the present disclosure may be implemented.DETAILED DESCRIPTION

[0042] The following detailed description illustrates embodiments of the present disclosure and ways in which they can be implemented. Although some modes of carrying out the present disclosure have been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practicing the present disclosure are also possible.

[0043] FIG. 1 illustrates a mining and / or construction machine 10 for rock bolting in a mining and / or construction environment. The mining and / or construction environment may be an underground environment. The mining and / or construction machine 10 may be adapted to operate autonomously or remotely controlled. For illustration purposes only, the mining and / or construction machine 10 is shown in connection with a borehole formed in a rock surface in the mining and / or construction environment. In some examples, the mining and / or construction machine 10 may be a rock bolting machine. The mining and / or construction machine 10 may comprise at least one boom 12, a drilling and bolting implement 14 coupled to the at least one boom 12, a detection system 16 comprising at least one movable imaging device 18, and a fluid injection arrangement 20 for injecting a fluid into the plurality of boreholes.

[0044] As shown in FIG. 1, the mining and / or construction machine 10 may also comprise a control device 100 for controlling operation of the mining and / or construction machine 10. The control device 100 comprises processing circuitry comprising at least one processor that is configured to execute computer-executable instructions to perform a method of controlling the mining and / or construction machine in accordance with examples of the present disclosure. As also shown in FIG. 1, the mining and / or construction machine 10 may communicate with a remote control system 200 that can also be referred to as a central controller. The remote control system 200 may be positioned in an operating room from which operations of work machines in the mining and / or construction environment may be controlled. In some examples, e.g., when the mining and / or construction machine 10 is operated fully or partially autonomously, the control device 100 may receive instructions from the remote control system 200 and the control device 100 may control operation of the mining and / or construction machine 10 based on these instructions.

[0045] The at least one boom 12 may comprise one or more booms. The drilling and bolting implement 14 may be coupled to the same boom or different booms of the at least one boom 12. A boom of the at least one boom 12 comprises several joints, and angles of the joints may be measured by a sensing system comprising one or more sensors suitable for the intended use, in some examples herein. For example, a slewing angle of a boom can be measured that refers to a horizontal rotation angle of the boom relative to a ground, allowing the boom to swing in a circular motion. A lift angle of the boom can also be measured, which refers to an angleDocket No.: DRS2993PC00formed between the boom and a horizontal plane such as e.g. a plane passing through the ground, measured in degrees. The boom also includes one or more turning mechanisms or devices, and rotation angles of the joints due to rotation of these devices may be measured by a sensing system comprising an angle sensor. In some examples, a boom may comprise prismatic or sliding joints that connect two machine components so that one components can move or slide linearly relative to the other while a rotation between the connected components is restricted. A suitable sensor may be used to measure an angle between a joint's sliding axis and a fixed reference line. In some examples, a length sensor may also be associated with the at least one boom, wherein the length sensor is configured to measure an elongation or extension of a feed or feeding mechanism relative to its holder by tracking a linear position of the feeding mechanism, to monitor and control with precision movements of the feeding mechanism. As used herein, a feed or feeding mechanism can refer to a tip of the fluid injection arrangement 20 from which the fluid is supplied or fed into the borehole. In some examples, a feed can refer to a forward motion of a drill end of the drilling and bolting implement 14.

[0046] The detection system 16 comprises the at least one movable imaging device 18, such as a first imaging device 18a and a second imaging device 18b shown in FIG. 1 by way of example. In some examples, the at least one imaging device 18 may comprise one or more digital cameras with Pan Zoom Tilt (PTZ) functionality. An imaging device of the at least one movable imaging device 18 may be attached at a location different from a boom, e.g., to a cabin 11 of the mining and / or construction machine 10, as shown for the first imaging device 18a in FIG. 1. The imaging device may be enclosed within a protective housing and may be associated with one or more lighting devices, to illuminate a target area. An imaging device of the at least one movable imaging device 18 may be attached to the boom 12, as shown for the second movable device 18b in FIG. 1. The detection system 16 may comprise multiple imaging devices, and one or more of the imaging devices may be controlled to acquire images that are used for determining a status of injection of fluid into a borehole as part of securing a bolt in the borehole.

[0047] It should be appreciated that the detection system 16 may comprise any suitable number of imaging devices, such as video cameras, and the imaging devices may be controlled to acquire images from a certain area of a rock surface and / or from the mining and / or construction machine 10 operating in a mining and / or construction environment. The detection system 16 can be completely or in part installed in the cabin 11 of the mining and / or construction machine 10 and some of the cameras may be positioned outside of the cabin 11. The techniques in accordance with the present disclosure may be used in connection with a detection system of any suitable type.

[0048] It should also be appreciated that the detection system 16 may comprise various other sensors, e.g., an audio sensor, a vibration sensor, a position sensor, a haptic feedback sensor, and / or any other sensors configured to acquire data regarding operation of the mining and / or construction machine 10.

[0049] The drilling and bolting implement 14 may be configured to drill a borehole in a planned borehole location and insert a rock bolt 26 into the drilled borehole. The drilling and bolting implement 14 may include at least one drill tool or drilling arrangement (not shown separately) at a distal end 14d of the drilling and bolting implement 14, the drill tool being configured to drill a borehole. The distal end 14d of the drilling and boltingDocket No.: DRS2993PC00implement 14, or a portion of the drilling and bolting implement 14 comprising the distal end 14d, may be referred to as an end-effector which is a working end that performs drilling and / or bolt installation. The drill tool may be a reusable drill tool such as a drill rod, a drill pipe, a drill bit, or a drill string. The rock bolt 26 may be inserted into the drilled borehole after the borehole is formed by the drilling and bolting implement 14.

[0050] In some examples, the drill tool may be a single-use tool such as, e.g., a hollow core bolt or a selfdrilling anchor bolt or a drill string. In such implementations, the drilling and bolting implement 14 may be configured to use a self-drilling anchor bolt 26 that is used for both drilling the borehole and reinforcing it. The rock bolt is secured in the borehole by using fluid injected by the fluid injection arrangement 20.

[0051] In addition, in some examples, no bolt may be used and the mining and / or construction machine 10 may be configured to inject the fluid directly into a drilled borehole.

[0052] The fluid injection arrangement 20 is configured and adapted to inject a fluid into a borehole 22 drilled in a body 24, such as a rock surface, in rock stabilizing applications. For this purpose, the fluid injection arrangement 20 may comprise an injector 31, a pump 32, and at least one guide or hose or conduit 33. As shown in FIG. 1, the borehole 22 comprises an interior borehole wall 23 and the body comprises a surrounding body wall 25 comprising a wall of the body 24 such as the rock surface. It should be appreciated that a shape of the interior borehole wall 23 may be uneven and that there may be cracks, cavities and other irregularities in the interior of the borehole 22. The body 24, such as a rock surface, may be porous. Thus, a volume of the borehole 22 may not be known or may be underestimated, which may complicate the process of injecting fluid in the borehole 22. The mining and / or construction machine 10 may be adapted to perform a rock bolting process, i.e. it may be adapted to autonomously drill the borehole 22, place a rock bolt 26, such as a separate rock bolt or a self-drilling anchor bolt, into the borehole 22, and inject a fluid into the borehole 22. The fluid may be a binding liquid, such as a resin-based grout material. The fluid may comprise at least two components mixed prior to injection, which harden through a chemical reaction, such as an exothermal chemical reaction.

[0053] In some examples, the mining and / or construction machine 10 may be configured to perform a rock bolting process involving a plurality of boreholes at planned borehole positions or locations. The mining and / or construction machine 10 may be configured to autonomously drill the borehole 22 at a planned borehole location, place a separate bolt or self-drilling anchor bolt 26 in the borehole 22, and inject the fluid into the borehole 22.

[0054] The detection system 16 may be arranged to be controlled, at least in part, to provide an appropriate view of a distal end 14d of the drilling and bolting implement 14 positioned at the location of the borehole 22, and of the fluid injection arrangement 20. It should be noted that the fluid injection arrangement 20 may be positioned completely or at least in part inside of the drilling and bolting implement 14. The mining and / or construction machine 10, e.g., the control device 100 in conjunction with the detection system 16, are configured to determine a status of injection of the fluid into the borehole by the fluid injection arrangement 20. The detection system 16 comprising the at least one movable imaging device 18 may acquire images or imaging data from an area where the drilling and bolting implement 14, such as a distal end 14d of the drilling and bolting implement 14, is positioned. The imaging data may be used to determine a status of injection of the fluid into the borehole by theDocket No.: DRS2993PC00fluid injection arrangement 20, as discussed in more detail below. For example, it may be detected as the fluid appears outside of the borehole.

[0055] FIG. 2 shows an enlarged view of the borehole 22 with the bolt 26 inserted therein. The detection system 16 as shown in FIG. 1 may be configured to determine a status of injection of the fluid into the borehole 22, which may involve detecting fluid outside of the borehole 22 and the fluid injection arrangement 20. For example, fluid may be detected at a borehole collar, i.e. at an intersection 27 between the borehole wall 23 and the surrounding body wall 25, or at portions of the fluid injection arrangement 20. A detection of fluid at the borehole collar may indicate that the status of injection of the fluid into the borehole 22 comprises the borehole 22 being successfully filled with fluid. A detection of fluid at portions of the fluid injection arrangement 20, e.g. at a connection point between the injector 31 and the bolt 26 or at other locations, may indicate that the status of injection of the fluid into the borehole 22 comprises detecting a leakage in the fluid injection arrangement 20. In some cases, the fluid may appear away from the distal end of the drilling and bolting implement 14.

[0056] FIGs. 3A, 3B, and 3C schematically illustrate one non-limiting example of a rock bolting process that can be performed by the mining and / or construction machine 10. In the example illustrated, the mining and / or construction machine 10 is configured to perform a whole rock bolting cycle, including drilling of the borehole 22 and injecting a fluid in the borehole 22. It should be noted that the fluid injection arrangement 20 can be at least in part positioned inside the drilling and bolting implement 14. In FIG. 3A, a drill or drill tool or drilling arrangement 15 of the drilling and bolting implement 14 comprises a bolt 26 which is a self-drilling anchor bolt in this example. The self-drilling anchor bolt 26 may comprise a drill bit at a tip or distal end thereof that comprises cutting edges that allow the bolt to drill a hole as it is advanced into material such as rock or other hard material. The drilling arrangement 15 is configured to drive the bolt 26 into the rock 24 thereby forming the borehole 22, as shown schematically by an arrow 35 in FIG. 3A. When the drilling of the borehole 22 is completed, the drilling arrangement 15 may be detached from the bolt 26 as shown schematically by an arrow 37 in FIG. 3B, and the fluid injection arrangement 20 may be connected to the bolt 26 as shown in FIG. 3C. The fluid injection arrangement 20 may be positioned in a suitable way for injection of the fluid into the borehole 22. The fluid injection arrangement 20 may then be activated or triggered to begin injecting fluid into the drilled borehole 22, as shown in FIG. 3C where the injector 31 having a distal end or tip 20d of the fluid injection arrangement 20 is shown to be positioned adjacent to the distal end 14d of the drilling and bolting implement 14. The self-drilling anchor bolt 26 may have a hollow through channel formed therethrough, and the fluid may thus be injected through the hollow channel.

[0057] In rock bolting, a challenge exists related to injection of a fluid into in a borehole - a volume of the borehole may be unknown and it may not be straightforward to determine when to control a mining and / or construction machine so that a fluid injection arrangement terminates injection of fluid into the borehole.Conventional approaches to rock bolting may include visual assessment by a human operator of a progress of the injection process, which may be error-prone due to low visibility in the underground mining and / or construction environment and the fact that the operator may be located remotely. Other existing approaches include using cartridges with a binding fluid, e.g. a resin, such as a predefined volume of the fluid is injected intoDocket No.: DRS2993PC00a borehole. This may however lead to securing a bolt in the borehole in an inadequate manner when the amount of the injected fluid is not sufficient. As another potential problem, leakages of the binding fluid and thus its waste may occur.

[0058] In general, in automated rock bolting, various processes, including drilling boreholes at planned locations and insertion of bolts into the grilled boreholes, may be performed autonomously. At the same time, determining a status of injection of a fluid into in a borehole remains to be a challenge. Techniques in accordance with the present disclosure provide an approach that advantageously allows monitoring a rock drilling and bolting process, as well as a fluid injection process, using imaging devices, and analyzing imaging data acquired by the imaging devices so that a status of injection of a fluid into in a borehole is determined with accuracy appropriate for autonomous and / or remotely controlled operations. The status may include a successful completion of filling of the borehole with a fluid, leakage of the fluid as part of an error or malfunction in the fluid injection arrangement, or another event.

[0059] FIG. 4 illustrates a process or method 400 for controlling operation of a mining and / or construction machine for drilling and rock bolting a plurality of boreholes in a rock surface. The mining and / or construction machine, e.g., mining and / or construction machine 10 shown in FIG. 1, comprises at least one boom, a drilling and bolting implement coupled to the at least one boom, a detection system comprising at least one movable imaging device, and a fluid injection arrangement for injecting a fluid into the plurality of boreholes. The drilling implement and the bolting implement may be mounted on the same boom or on different booms - e.g. a drill tool or mechanism may be coupled to one boom whereas a bolting tool or mechanism may be coupled to a different bolt. The detection system may be installed in the cabin of the mining and / or construction machine, or in some cases in other parts of the mining and / or construction machine.

[0060] The method 400 may be performed by a suitable control system, controller, or control device such as, e.g., control device 100 shown in FIG. 1. The control device 100, which may be configured, e.g., as shown in FIGs. 6A and 6B, comprises at least one processor configured to perform the method 400. The control device 100 may be positioned on the mining and / or construction machine 10 as shown in FIG. 1 and in FIGs. 2, 3A, 3B, and 3C. In some examples, e.g., when the mining and / or construction machine 10 is controlled to operate autonomously, the control device 100 may be in operative communication with a remote control system or device. For example, a remote control system 200 as shown in FIG. 1, which can also be referred to as a central controller, may be positioned in an operating room from which operations of work machines in the mining and / or construction environment may be controlled. The remote control system 200 allows teleoperation or remote control of the mining and / or construction machine 10, such that the mining and / or construction machine 10, e.g., the control device 100, may receive operating instructions for controlling operation of the mining and / or construction machine 10 from the remote control system 200. The control device 100 can send to the remote control system 200 data acquired by various sensors and systems, e.g., by detection system 16 and other systems for tracking operation of the mining and / or construction machine 10.

[0061] The process or method 400 may be initiated at a suitable time - for example, as the mining and / or construction machine begins operating to perform drilling and rock bolting the plurality of boreholes in the rockDocket No.: DRS2993PC00surface. The method 400 is performed to automatically monitor or determine a status of injection of the fluid into a borehole by the fluid injection arrangement, to control the fluid injection arrangement in response to the determined status of injection of the fluid.

[0062] At block 402, the method 400 may optionally comprise determining a position and angle of one or more joints of the boom having the drilling and bolting implement coupled thereto. The drilling implement and the bolting implement may be mounted on the same boom or on different booms. As discussed above, a slewing angle of a boom can be measured that refers to a horizontal rotation angle of the boom relative to a ground, allowing the boom to swing in a circular motion. A lift angle of the boom can also be measured, which refers to an angle formed between the boom and a horizontal plane such as e.g. a plane passing through the ground, measured in degrees. A length sensor may also be used that is configured to measure an elongation or extension of a feed or feeding mechanism, e.g., a tip of the fluid injection arrangement 20 from which the fluid is supplied or fed into the borehole, relative to its holder by tracking a linear position of the feeding mechanism. Information acquired by others sensors may be used as well.

[0063] At block 404, the method comprises obtaining first imaging data acquired by the at least one movable imaging device from a first portion of the rock surface including a planned borehole location, wherein one borehole is to be drilled approximately at the planned borehole location. The one borehole refers to a borehole that is to be drilled. The first portion of the rock surface encompasses the planned borehole location and may include a relatively wide area around the planned borehole location. The first portion may be defined as an area that would typically be imaged by one or more imaging devices of the mining and / or construction machine 10, but imaging the first portion would not be sufficient for determining a status of injection of the fluid into the one borehole. The imaging device may be positioned to image at least a portion of the drilling and bolting implement 14.

[0064] At block 406, the method comprises identifying, within the first portion of the rock surface, based at least on the planned borehole location and on the first imaging data, a focus region encompassing a position of a distal end of the drilling and bolting implement at the planned borehole location. The first imaging data may be analyzed to identify the position of the distal end of the drilling and bolting implement at the planned borehole, and in some cases other, more proximal portions of the drilling and bolting implement. In this way, the focus region may include a distal end of the fluid injection arrangement and other, more proximal portions of the fluid injection arrangement. As mentioned above, the distal end of the fluid injection arrangement may be not visible due to the fluid injection arrangement being positioned, at least in part, inside the drilling and bolting implement. Thus, monitoring the distal end of the drilling and bolting implement at the planned borehole location may allow monitoring the status of the injection of the fluid, which involves detecting whether the fluid appears outside of the borehole.

[0065] In some examples, the focus region may be identified or selected such that the distal end of the drilling and bolting implement 14 is positioned in a center or approximately in the center of the focus region.

[0066] In some examples, when the focus region is identified, the movable imaging device may be controlled to move only if the drilling and bolting implement moves, such that the imaging device follows the distalDocket No.: DRS2993PC00end of the drilling and bolting implement. A zoom of the imaging device may be adjusted automatically, to ensure that a fluid can be identified and located in positions at a collar of the borehole and at a distance from the collar of the borehole. The binding fluid may be detected at different locations relative to the borehole.

[0067] The focus region may be smaller than the first portion of the rock surface, such that the imaging device may be zoomed in onto the focus region. The focus region may be identified so that the at least one movable imaging device may be controlled to acquire imaging data from the focus region, to determine a status of injection of the fluid into the one borehole.

[0068] In some examples, identifying the focus region at block 406 may comprise identifying the focus region further based on the position and angle of one or more joints of the at least one boom having the drilling and bolting implement coupled thereto. The position and angle of one or more joints of the boom may be determined at block 402.

[0069] In some examples, borehole location, such as borehole coordinates, indicating a planned location for the borehole may be directly used to aim at least one movable imaging device into a required position, given known position of the mining and / or construction machine 10 relative to the borehole coordinates. The borehole coordinates may be specified relative to a defined reference, such as, e.g., a local survey point defined for the area in which the mining and / or construction machine is operating, a three dimensional (3D) grid applied to the rock surface, or a global coordinate system e.g. the global positioning system (GPS).

[0070] At block 408, the method comprises controlling the at least one movable imaging device to acquire second imaging data from the focus region. The focus region, which also referred to as a region of interest, may include a portion of the rock surface and at least the distal end of the drilling and bolting implement. In this way, the focus region also encompasses a tip or distal end of the fluid injection arrangement. In some examples, the second imaging data may at least partly overlap with the first imaging data.

[0071] The at least one movable imaging device, e.g., one or more movable imaging devices, may be directed onto the focus region to acquire the second imaging data. For example, one or more movable imaging devices from the at least one movable imaging device can be controlled to acquire the second imaging data from the focus region. In some cases, the at least one movable imaging device may be controlled to keep its focus, or otherwise to follow, an area around the borehole where appearance of the fluid outside of the borehole is expected. Also, the at least one movable imaging device may in some cases be controlled to repeatedly change the focus region, e.g., by zooming out and zooming in onto the drilling and bolting implement and / or fluid injection arrangement, to be able to detect the fluid at a position that is offset from the location of the borehole.

[0072] The at least one movable imaging device may be directed to image the focus region while ensuring a proper spatial relationship between the imaging device, the distal end of the drilling and bolting implement, and the focus region, to ensure an accurate visual alignment and image clarity. The at least one movable imaging device may be positioned at a certain offset relative to a drilling axis of the drilling and bolting implement, such that an optical axis of the imaging device can intersect the planned borehole location. The focusing may be achieved by an automated focusing mechanism associated with or included in the at least one movable imagingDocket No.: DRS2993PC00device, which can be configured to operate responsive to image feedback. In this way, a clear, high-resolution view of the focus region may be achieved during a drilling and / or bolting operation.

[0073] In some examples, the method 400 may comprise selecting an imaging device of the at least one movable imaging device to acquire the second imaging data from the focus region. The mining and / or construction machine may comprise multiple imaging devices, and the imaging device, e.g., one or more imaging devices, may be selected for imaging the focus region. In some examples, different imaging device(s) may be used for acquiring the first imaging data and the second imaging data, respectively.

[0074] In some examples, the fluid injection arrangement may be controlled to begin injecting the fluid into the one borehole after the second imaging data is analyzed to determine if there are one or more surface elements in the focus region. The surface elements, if identified, are excluded from consideration when analyzing the second imaging data to determine a status of injection of the fluid into the one borehole. In this way, the status of injection of the fluid into the one borehole is determined with improved precision and accuracy. The status of the injection of the fluid into the one borehole may be determined on an ongoing basis, in real time, as the second imaging data is acquired while the fluid injection arrangement is injecting the fluid into the drilled borehole.

[0075] In some examples, controlling the at least one movable imaging device to acquire the second imaging data from the focus region is performed in response to initiating the fluid injection arrangement to begin injecting the fluid into the one borehole. Thus, once the focus region is identified and the fluid injection arrangement is initiated to begin injecting the fluid into the one borehole, the at least one movable imaging device may be triggered to begin acquiring the second imaging data from the focus region. In this way, a status of the injection of the fluid into the borehole may be determined in real time, including detecting when the fluid appears outside of the borehole.

[0076] In some examples, a metal bolt, such as e.g. a steel bolt, may be inserted by the drilling and bolting implement into the drilled borehole to stabilize the rock surface, such as e.g. a mine's roof or wall. In some examples, the bolt may be a so-called self-drilling anchor that may be used by the drilling and bolting implement for drilling and bolting, such that the bolt itself is used to perform the drilling. Regardless of the specific configuration of the bolt and the drilling and bolting implement, in some examples, the fluid injection arrangement may be triggered or initiated to begin injecting the fluid into the one borehole after the one borehole has been drilled. The fluid, such as, e.g., resin, is injected to fill the space between the bolt and the interior of the borehole to thereby anchor the bolt in the borehole. In implementations in which a self-drilling anchor is used, such anchor may have a hollow channel formed therethrough, and the resin may be injected into the hollow channel.

[0077] At block 410, the method comprises analyzing the second imaging data to determine a status of injection of the fluid into the one borehole, which injection is performed by the fluid injection arrangement. The status of injection of the fluid into the one borehole may comprise fluid outside of the one borehole or leakage in the fluid injection arrangement. Detection of fluid outside of the one borehole may be indicative of the borehole being filled such that continuing fluid injection would be wasteful. In some cases, the fluid outside of the one borehole may be indicative of leakage, e.g., due to malfunction, in the fluid injection arrangement.Docket No.: DRS2993PC00

[0078] In some examples, analyzing the second imaging data to determine the status of injection of the fluid into the one borehole comprises applying a machine-learning (ML) model to the second imaging data. Thus, the processing at block 410 may include obtaining the ML model that can be applied to the second imaging data to determine the status of injection of the fluid into the one borehole. In some examples, the ML model may comprise at least one deep learning model. The deep learning model may be deployed on the mining and / or construction machine. A deep learning model may learn representations directly from raw data, performs automatic feature selection, and may achieve higher performance than conventional approaches to image processing. The deep learning model may be trained, e.g., as shown for a method 500 illustrated in FIG. 5. In some examples, the at least one deep learning model may comprise a convolutional neural network (CNN) model. The CNN model may comprise VGG, LeNet, AlexNet, ResNet, ResNet, EfficientNet, MobileNet, U Net, the like, or combinations thereof. In some examples, the at least one deep learning model comprises a semantic segmentation model.

[0079] The second imaging data may be preprocessed before applying the deep learning model to analyze the data. For examples, the images can be normalized and corrections for potential lens distortion may be made. The acquired second imaging data may be augmented to account for brightness / contrast, motion blur e.g. due to fog at the borehole location, dust / particle overlays, random glare and other factors.

[0080] The deep learning model may be trained to classify or segment image regions within the second imaging data, to detect the fluid from the second imaging data, which is indicative in the fluid outside of the one borehole. In some examples, the training data may be labeled, such as a supervised deep learning model may be used. In some examples, an unsupervised deep learning model may be used. The use of the deep learning model may allow achieving higher accuracy, robustness, and adaptability than when conventional image analysis methods are used.

[0081] At block 411, the method 400 may comprise analyzing the second imaging data to determine whether the focus region comprises a surface element in the focus region, wherein the surface element may be excluded from consideration when analyzing, at block 410, the second imaging data to determine a status of injection of the fluid into the one borehole. The processing at block 411 may be performed as part of the processing at block 410, as shown schematically in FIG. 4.

[0082] In some examples, the processing at block 411 may be performed before the fluid injection arrangement is activated or initiated to begin injection of the fluid into the drilled borehole. Thus, in some examples, the method may comprise initiating the fluid injection arrangement to begin injecting the fluid into the one borehole after the second imaging data is analyzed to determine whether the focus region comprises a surface element in the focus region. The surface element may be excluded from consideration when analyzing the second imaging data to determine the status of injection of the fluid into the one borehole. The method in accordance with examples of the present disclosure allows determining, using image processing, objects in the focus region such as various surface features, and disregarding those objects when determining the status of injection of the fluid into the one borehole, which injection is performed by the fluid injection arrangement. In thisDocket No.: DRS2993PC00way, the monitoring of the fluid injection process and determining its status is performed with improved precision and accuracy.

[0083] In some examples, the surface element may be at least one selected from the group consisting of a previously installed bolt part, a residue from a previous fluid injection, and a geological feature in the focus region. The residue from the previous fluid injection may have a different color, brightness, and / or other characteristics as compared to characteristics of a freshly injected fluid, such as resin, which is detected using the techniques in accordance with the present disclosure. The previously installed bolt part and a geological feature may also have different characteristics as compared to characteristics of the freshly injected fluid. The geological feature may be a crack in the rock surface, a material of a certain type, and others. Other types of surface elements may be detected within the focus region, which are elements different from the fluid being injected into the one borehole. The surface element may be adjacent to the planned borehole location where the one borehole is being drilling or has been drilled. In some examples, the method may discern which types of surface elements are included in the focus region. The surface elements may be detected by applying a machine-learning model, such as a deep learning model, to the second imaging data, as discussed in more detail below.

[0084] Thus, analysis of the second imaging data may involve determining whether there are features or surface elements in the rock surface, which may potentially interfere with determining a status of injection of the fluid into the one borehole. The surface element, if not detected and recognized using the techniques in accordance with the present disclosure, may be mistaken for the fluid, either by an operator or automatically by the control device, and detecting and recognizing the surface element may therefore improve accuracy of the analysis of the second imaging data to determine the status of injection of the fluid into the one borehole. In some examples, determining that the focus region comprises a surface element may be used in assessing conditions of the rock surface, and the information on presence or absence of the surface element may be recorded for future use.

[0085] Furthermore, in some examples, the method 400 may comprise analyzing the second imaging data to determine whether the focus region comprises fog, dust, and other elements that may interfere with reliable analysis of the second imaging data to determine a status of injection of the fluid into the one borehole.

[0086] Accordingly, techniques in accordance with the present disclosure allow determining a status of injection of the fluid into the borehole with increased accuracy.

[0087] In some examples, the method 400 may comprise, as the second imaging data is being analyzed to determine the status of injection of the fluid into the one borehole, dynamically adjusting the focus region based at least on one or more characteristics of an environment in the focus region. In some examples, the focus region is selected such that the end-effector or the distal end of the drilling and bolting implement 14 is positioned in a center or approximately in the center of the focus region. The end-effector may be pointing straight up relative to the rock surface. In some examples, controlling the mining and / or construction machine to maintain the endeffector in the center-top of the focus region may allow timely detection of fluid as it exits the borehole and drips down.Docket No.: DRS2993PC00

[0088] The one or more characteristics of the environment in the focus region may be fog, dust, pieces of rock, lighting changes and insufficient light, and other features that may affect accurate detection of the status of injection of the fluid into the one borehole. The one or more characteristics of the environment in the focus region may be detected using the deep learning model as discussed above. Dynamically adjusting the focus region may involve dynamically adjusting a size and location of the focus region. In some examples, for example, when the fog is present around the distal end of the drilling and bolting implement, and / or along a more proximal portion of the drilling and bolting implement, the at least one movable imaging device may be controlled to zoom out, such that a larger and / or an alternative area may be imaged as part of acquiring the second imaging data. The at least one movable imaging device may be controlled to zoom in on the area around the distal end of the drilling and bolting implement. In some examples, the at least one movable imaging device may be controlled to repeatedly zoom out and then zoom in on the area around the distal end of the drilling and bolting implement. In some examples, the fluid may appear not at the location of the borehole, but further away from the borehole, at a more proximal location along the drilling and bolting implement. The dynamic adjustment of the focus region may allow accurately detecting the appearance of the fluid outside of the borehole, even if the fluid appears further away from the borehole.

[0089] In some examples, the status of injection of the fluid into the borehole, which is also referred to herein as the one borehole, may be determined in conjunction with other techniques. For example, an approach can be used as described in International Patent Application No. PCT / SE2024 / 050974, filed on 14 November 2024, and entitled "METHOD FOR INJECTING FLUID INTO A BOREHOLE, CONTROL DEVICE, SYSTEM AND A MINING AND / OR CONSTRUCTION MACHINE,” which is incorporated by reference herein in its entirety. For example, the method 400 may additionally consider a timing of when the fluid is detected outside of the borehole. Thereby, the method may distinguish between an expected detection of fluid when the borehole is sufficiently filled, and a leakage occurring at the fluid injection arrangement. Thus, the method may take advantage of the fact that an expected detection of fluid when the borehole is sufficiently filled, i.e. indicating a successful injection, may occur after a certain time period from the start of the injection, whereas a leakage may occur immediately or shortly after the initiation of the fluid injection. The information on the timing of when the fluid is detected in the second imaging data may be used in conjunction with the assessment based on an image processing analysis.

[0090] In some examples, a maximum time period or duration may be used, which defines a time that the fluid injection arraignment may be allowed to operate from its activation for injecting the fluid. Upon expiration of the time period, the fluid injection arrangement may be controlled to stop or terminate the fluid injection. This time period may be set automatically or based on operator input. The operator may be controlling the mining and / or construction machine remotely.

[0091] At block 412, the method comprises controlling the fluid injection arrangement in response to the determined status of injection of the fluid. In some examples, determining the status of injection of the fluid comprises detecting fluid outside of the one borehole, and the method comprises, in response to the detection of the fluid outside of the one borehole, controlling the fluid injection arrangement to terminate the injection of theDocket No.: DRS2993PC00fluid into the one borehole. In some examples, determining the status of the injection of the fluid comprises detecting a leakage in the fluid injection arrangement, and the method comprises, in response to the detection of the leakage, controlling the fluid injection arrangement to terminate the injection of fluid into the one borehole and generate an error signal. In this way, the fluid injection arrangement may be automatically controlled to abort or stop or terminate the fluid injection. The error signal may be error information or instruction in any suitable format. The error signal may be generated in conjunction with an action that can be suggested or automatically initiated in response to the detection of the leakage that may be indicative of the fluid injection not being successful.

[0092] In some examples, the method comprises determining the status of the injection of the fluid which is indicative of a need to adjust one or more parameters of the fluid injection process. Thus, the method may comprise controlling the fluid injection arrangement to adjust one or more parameters of the fluid injection process performed by the fluid injection arrangement.

[0093] The mining and / or construction machine may be controlled to take other physical actions in response to the determined status of injection of the fluid. In some examples, in response to detecting a leakage in the fluid injection arrangement, indicating an irregularity in operation of the fluid injection arrangement, the control device 100 may generate an alert, alarm, or error information or message, e.g., to an operator of the mining and / or construction machine, to the mining and / or construction machine, to the remote control system 200, and / or to another system or device. The error information may be in a visual, acoustic format, or a combination thereof. In some examples, the error information may be generated and provided in conjunction with an instruction related to an action to be taken in response to the detected fluid leakage.

[0094] The operator may be controlling and / or monitoring operation of the mining and / or construction machine remotely. The error information may automatically trigger an alert, alarm, or error information and / or a stop of the fluid injection operation. In some cases, the error information may be displayed to an operator on a suitable user interface, e.g., a user interface associated with the remote control system 200. Information on the error may be recorded in a suitable storage media.

[0095] The processing in accordance with the method 400 may be performed for multiple boreholes of the plurality of boreholes planned to be drilled in the rock surface, e.g., in accordance with a drill plan. Thus, the processing at processing blocks of FIG. 4 may be repeated for each borehole of the plurality of boreholes.

[0096] Accordingly, a rock bolting process in accordance with examples of the present disclosure is provided, which may achieve a reliable, automatic determination of the status of the injection of the fluid. The method may achieve a rock bolting process which ensures that the boreholes are sufficiently filled and which provides for an improved detection of irregularities during the injection, such as leakages from the fluid injection arrangement. In this way, reliance on a human operator for determining the status of the injection of the fluid by the fluid injection arrangement is reduced or eliminated. As another advantage, a risk of waste of the fluid is reduced or eliminated, thereby costs of operating of the mining and / or construction machine are reduced.

[0097] FIG. 5 illustrates an example of a process or method 500 of training and a deep learning model to the second imaging data, in accordance with examples of the present disclosure. The at least one deep learning model may comprise a convolutional neural network (CNN) model or another suitable deep learning model. TheDocket No.: DRS2993PC00process or method 500 may be performed by a suitable control system or computer system. In some examples, the deep learning model that can be used in the method in accordance with techniques of the present disclosure may be generated on a remote computer system, e.g., at the remote control system 200 or at another controller or control system.

[0098] At block 502, the method 500 comprises acquiring training data, for example, data acquired from imaging a rock surface, such as the rock surface in which the borehole is drilled, or other one or more rock surfaces. The training data may include images of parts of a mining and / or construction machine, e.g., of a drilling and bolting implement, a fluid injection arrangement, and other parts that can be within a field of view of the imaging device during imaging for determining a status of injection of a fluid into a borehole in accordance with examples of the present disclosure. In some examples, the deep learning model may be trained using simulation data. The training data may include images including various surface features, e.g., of previously installed bolt parts, of a residue from a previous fluid injection, and geological features that may be present in a surface in which boreholes are drilled.

[0099] In examples in accordance with the present disclosure, a supervised deep learning model may be used. The training data, e.g., images, are labeled as images including a fluid detected outside of the borehole, images including one or more surface elements and other elements, etc. The trained data may thus comprise labelled image data representing instances of fluid appearance or leakage due to successful filling of a borehole, fluid leakage due to unsuccessful filling of a borehole e.g. due to leakage in the fluid injection arrangement, and fluid leakage due to other conditions. In some examples, the training data includes images of a rock surface including fluid such as resin and the end-effector of the drilling and bolting implement 14, wherein the endeffector is positioned at a distance from the resin. The images used for training may include images acquired under various different lighting conditions, different angles, different dust levels, for different types of equipment, and various rock types. Also, one or more types of fluid may be represented on the images, with different colors, different times when the fluid was applied e.g. to secure a previously installed bolt, different brightness, texture, and other characteristics.

[0100] At block 504, the method 500 comprises selecting the deep learning model. The deep learning model may be selected based on a required robustness and accuracy, and inference speed, for real-time use, of the detection of the status of the injection of the fluid and other features. The model may be selected based on whether it will be deployed on the mining and / or construction machine 10 or on the remote control system 200, based on a type of mining and / or construction machine, and other factors.

[0101] In some examples, the at least one deep learning model may comprise an object detection network model, e.g., YOLO-NAS, YOLOv8, Faster R-CNN, or a Single Shot Detector (SSD). YOLO ("You Only Look Once”) is an object detection architecture that can be used to identify and locate objects in an image or video using a single pass through a neural network. YOLO-NAS refers to a YOLO Neural Architecture Search (NAS), and YOLOv8 is an extension of YOLO. The Faster R-CNN relies on a first stage - a Region Proposal Network (RPN) that is a neural network that generates a list of candidate regions containing objects, which is then fed to a second stage for classification and bounding box refinement. In some examples, different YOLO models may beDocket No.: DRS2993PC00used and a suitable YOLO model may be selected based on its performance such as accuracy, speed, and robustness. Other types of deep learning models, suitable for identifying and positioning a fluid injected as part of the drilling and bolting process, may be used in addition or alternatively.

[0102] In some examples, the at least one deep learning model may comprise a semantic segmentation network model, e.g., U-Net, DeepLabV3+, or SegFormer. The semantic segmentation network model may be able to classify every pixel in an image into a predefined category, whereby enabling the control device to analyze the second imaging data at a pixel level.

[0103] In some examples, the at least one deep learning model may comprise a video-based or a temporal deep network model e.g., ConvLSTM, 3D CNN, or a transformer-based video model. The video-based or a temporal deep network model may be useful for detecting and handling transient conditions, such as, e.g., fog, dust, small pieces of rock, and lighting changes, which is useful in determining conditions in the mine in real time.

[0104] At block 506, the method 500 comprises training the selected deep learning model using the acquired training data. The training of the selected model may include applying augmentations, e.g., motion blur, noise, dust occlusion, etc. Training the deep learning model involves teaching the deep learning model to recognize the fluid by learning from the labeled training data comprising imaging data. The deep learning model may be trained on a computer system or device that is separate from the mining and / or construction machine 10, e.g., at the remote control system 200 or at another control system or computer system.

[0105] At block 508, the method 500 may comprise evaluating the deep learning model. In some examples, the evaluation may be performed on imaging data which was not used in training the deep learning model. The model evaluation may be performed to determine if the model can perform with sufficient reliability, accuracy, and robustness. Any suitable approach may be used to evaluate the deep learning model. It should be noted that, in some examples, the processing at blocks 504, 506, and 508 of FIG. 5 may be performed at least partially simultaneously.

[0106] The trained and evaluated deep learning model may be updated and thereby improved as further data is acquired and more information is gathered about automatic assessment of a status of fluid injection into a borehole by the fluid injection arrangement. Over time, as the deep learning model is fed with more data, it improves its ability to accurately recognize the fluid in real time, as the fluid appears outside of the borehole in the focus region, including in cases where the fluid appears in previously unseen settings and / or at different locations relative to the borehole. The deep learning model may be updated and an updated deep learning model may be received by the control device 100 of the mining and / or construction machine 10.

[0107] The method steps described herein may be performed by a controller, control device or a control system e.g. control device 100 of the mining and / or construction machine 10, the remote control system 200, and / or any other suitable control device. The method steps may be performed by more than one of these controllers or computer systems, or by another suitable controller configured to control operation of a mining and / or construction machine in a mining and / or construction environment, the mining and / or construction machine comprising at least one boom, a drilling and bolting implement coupled to the at least one boom, a detection system comprising at least one movable imaging device, and a fluid injection arrangement for injectingDocket No.: DRS2993PC00a fluid into the plurality of boreholes. An example of an arrangement of a controller or control system 600, representing any of these controllers or control systems, is shown in FIGs. 6A and 6B.

[0108] As shown in FIG. 6A, the control system 600 may comprise an input and output interface 601 comprising communication circuitry configured to communicate with any components of a device on which the control system 600 is positioned or with which the control system 600 is otherwise associated, and with external devices and components. For example, if the control system 600 comprises the control device 100 of the mining and / or construction machine 10, the input and output interface 601 may be configured to communicate with one or more of the drilling and bolting implement 14, the detection system 16 comprising at least one movable imaging device, the fluid injection arrangement 20, and with the remote control system 200. If the control system 600 comprises the remote controller or control system 200, the input and output interface 601 may be configured to communicate with one or more mining and / or construction machines, e.g. the mining and / or construction machine 10, such as with the control device 100.

[0109] The input and output interface 601 may be configured to receive sensor measurements, send commands or instructions or signals to the drilling and bolting implement 14, the detection system 16 comprising at least one movable imaging device, the fluid injection arrangement 20 and other components, devices, or systems. The input and output interface 601 may be configured to enable the control device 100 to communicate with remote systems and / or controllers such as, e.g., the remote control system 200 in a mine control or operating room from which operations of work machines in the mining and / or construction environment can be controlled.

[0110] The input and output interface 601 may be configured to receive and send any other information, data or signals. The input and output interface 601 may comprise a wireless and / or wired receiver, a wireless and / or wired transmitter, and / or wireless and / or wired transceiver. The control system 600 may use the input and output interface 601 to control, operate and communicate with sensors, actuators, subsystems, and user interfaces of the mining and / or construction machine 10 and other devices by using any one or more out of a Controller Area Network (CAN), ethernet cables, Wi-Fi, Bluetooth, and other network interfaces including interfaces for communicating via wireless communications networks. The control system 600 may be configured to communicate with one or more external services, databases, and / or controllers via wireless communications technologies.

[0111] The method described herein may be implemented through a processing circuitry, e.g., at least one processor 602 of the control system 600 depicted in FIG. 6A, together with computer program code for performing the functions, actions, and steps of the embodiments herein. The computer program code may be provided as a computer program medium, for instance in the form of a tangible non-transitory computer-readable storage medium storing computer program code or computer-executable instructions for performing the method in accordance with examples of the present disclosure when loaded into the control system 600 and executed by the at least one processor 602. An example of a computer-readable storage medium may be in the form of a memory stick or any other appropriate medium that can hold machine readable data. The computer program code may furthermore be provided as a program code on a server and downloaded to the control system 600.Docket No.: DRS2993PC00

[0112] The control system 600 may comprise hardware memory 604 comprising one or more memory units. The memory 604 is configured to store computer-executable instructions executable by the at least one processor 602 of the control system 600. The memory 604 is arranged and configured to be used to store the computer-executable instructions, information, data, etc., to perform the method in accordance with examples of the present disclosure. The control system 600 may obtain information from an external memory.

[0113] The control system 600 may comprise one or more units that are configured to control the mining and / or construction machine 10, e.g., the drilling and bolting implement 14, the detection system 16, the fluid injection arrangement 20 and other components, devices, or systems, in accordance with examples of the present disclosure.

[0114] As shown in FIG. 6B, the control system 600 may comprise a determining unit 606. The control system 600, the at least one processor 602, and / or the determining unit 606 may be configured to determine a position and angle of one or more joints of the at least one boom having the drilling and bolting implement coupled thereto.

[0115] The control system 600 may comprise an obtaining unit 608. The control system 600, the at least one processor 602, and / or the obtaining unit 608 are configured to obtain first imaging data acquired by the at least one movable imaging device from a first portion of the rock surface including a planned borehole location, wherein one borehole is to be drilled approximately at the planned borehole location. The obtaining may be performed as described e.g. in connection with block 404 of FIG. 4.

[0116] As also shown in FIG. 6B, the control system 600 may comprise an identifying unit 610. The control system 600, the at least one processor 602, and / or the identifying unit 610 are configured to identify, within the first portion of the rock surface, based at least on the planned borehole location and on the first imaging data, a focus region encompassing a position of a distal end of the drilling and bolting implement at the planned borehole location. The controlling may be performed as described e.g. in connection with block 406 in FIG. 4. In some examples, identifying the focus region may be further based on the position and angle of one or more joints of the boom.

[0117] The control system 600 may comprise a controlling unit 612. The control system 600, the at least one processor 602, and / or the controlling unit 612 are configured to control the at least one movable imaging device to acquire second imaging data from the focus region. The controlling may be performed as described e.g. in connection with block 408 of FIG. 4. In some examples, controlling the at least one movable imaging device to acquire the second imaging data from the focus region may be performed in response to initiating the fluid injection arrangement to begin injecting the fluid into the one borehole. In some examples, the fluid injection arrangement may be controlled to begin injecting the fluid into the one borehole after the second imaging data is analyzed to determine if there are one or more surface elements in the focus region. The surface elements, if identified, are excluded from consideration when analyzing the second imaging data to determine a status of injection of the fluid into the one borehole. The control system 600 may comprise an analyzing unit 614. The control system 600, the at least one processor 602, and / or the analyzing unit 614 are configured to analyze the second imaging data to determine a status of injection of the fluid into the one borehole, which injection isDocket No.: DRS2993PC00performed by the fluid injection arrangement. The analyzing may be performed as described e.g. in connection with block 410 of FIG. 4.

[0118] In some examples, the analyzing unit 614 may comprise or may be an image processing unit. In some examples, the image processing unit may include a machine learning module 615 as shown in FIG. 6B. The machine learning module 615 may be used to apply a machine-learning model to imaging data, such as e.g. at least one deep learning model as described, e.g., in connection with FIG. 5. The machine-learning model may be deployed on the mining and / or construction machine 10, e.g. on the control device 100.

[0119] Thus, in some examples, analyzing the second imaging data to determine the status of injection of the fluid into the one borehole may comprise applying, e.g., by the machine learning module 615, a machinelearning model to the second imaging data. In some examples, the machine-learning model comprises at least one deep learning model. In some examples, the at least one deep learning model may comprise a CNN model. Other types of deep learning models may be used to analyze the second imaging data.

[0120] In some examples, the control system 600, the at least one processor 602, and / or the analyzing unit 614 may be configured to analyze the second imaging data to determine whether the focus region comprises a surface element in the focus region, wherein the surface element is excluded from consideration when analyzing the second imaging data to determine a status of injection of the fluid into the one borehole. The surface element may be at least one selected from the group consisting of a previously installed bolt part, a residue from a previous fluid injection, and a geological feature in the focus region.

[0121] The control system 600, the at least one processor 602, and / or the controlling unit 612 are further configured to control the fluid injection arrangement in response to the determined status of injection of the fluid. The controlling may be performed, as shown e.g. in connection with block 412 of FIG. 4.

[0122] In some examples, determining the status of injection of the fluid comprises detecting fluid outside of the one borehole, and the method comprises, in response to the detection of the fluid outside of the one borehole, controlling the fluid injection arrangement to terminate the injection of the fluid into the one borehole. In some examples, determining the status of the injection of the fluid comprises detecting a leakage in the fluid injection arrangement, and the method comprises, in response to the detection of the leakage, controlling the fluid injection arrangement to terminate the injection of fluid into the one borehole and generate an error signal. In this way, the fluid injection arrangement may be automatically controlled to terminate the injection of the fluid into the one borehole.

[0123] Those skilled in the art will appreciate that the units in the control system 600 described above may refer to a combination of analogue and digital circuits, and / or one or more processors configured with software and / or firmware, e.g., stored in the control system 600, that, when executed by the respective one or more processors, may perform the methods in accordance with embodiments of the present disclosure. One or more of these processors, as well as the other digital hardware, may be included in a single Application-Specific Integrated Circuitry (ASIC), or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a system-on-a-chip.Docket No.: DRS2993PC00

[0124] The operational steps described in any of the exemplary aspects herein are described to provide examples and discussion. The steps may be performed by hardware components, may be embodied in machineexecutable instructions to cause a processor to perform the steps, or may be performed by a combination of hardware and software. Although a specific order of method steps may be shown or described, the order of the steps may differ. In addition, two or more steps may be performed concurrently or with partial concurrence.

[0125] The terminology used herein is for the purpose of describing particular aspects only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including” when used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0126] It will be understood that, although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the scope of the present disclosure.

[0127] Relative terms such as “below” or “above” or “upper” or “lower” may be used herein to describe a relationship of one element to another element as illustrated in the Figures. It will be understood that these terms and those discussed above are intended to encompass different orientations of the device in addition to the orientation depicted in the Figures. It will be understood that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or intervening elements may be present. In contrast, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present.

[0128] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0129] It is to be understood that the present disclosure is not limited to the aspects described above and illustrated in the drawings; rather, the skilled person will recognize that many changes and modifications may be made within the scope of the present disclosure and appended claims. In the drawings and specification, there have been disclosed aspects for purposes of illustration only and not for purposes of limitation, the scope of the inventive concepts being set forth in the following claims.

Claims

Docket No.: DRS2993PC00CLAIMSWhat is claimed is:

1. A method (400) for controlling operation of a mining and / or construction machine for drilling and rock bolting a plurality of boreholes in a rock surface, the mining and / or construction machine comprising at least one boom, a drilling and bolting implement coupled to the at least one boom, a detection system comprising at least one movable imaging device, and a fluid injection arrangement for injecting a fluid into the plurality of boreholes, the method comprising:obtaining (404) first imaging data acquired by the at least one movable imaging device from a first portion of the rock surface including a planned borehole location, wherein one borehole is to be drilled approximately at the planned borehole location;identifying (406), within the first portion of the rock surface, based at least on the planned borehole location and on the first imaging data, a focus region encompassing a position of a distal end of the drilling and bolting implement at the planned borehole location;controlling (408) the at least one movable imaging device to acquire second imaging data from the focus region;analyzing (410) the second imaging data to determine a status of injection of the fluid into the one borehole, which injection is performed by the fluid injection arrangement; andcontrolling (412) the fluid injection arrangement in response to the determined status of injection of the fluid.

2. The method of claim 1, comprising analyzing (411) the second imaging data to determine whether the focus region comprises a surface element in the focus region, wherein the surface element is excluded from consideration when analyzing (410) the second imaging data to determine a status of injection of the fluid into the one borehole.

3. The method of claim 2, wherein the surface element is at least one selected from the group consisting of a previously installed bolt part, a residue from a previous fluid injection, and a geological feature in the focus region.

4. The method of claim 2 or claim 3, comprising initiating the fluid injection arrangement to begin injecting the fluid into the one borehole after the second imaging data is analyzed to determine whether the focus region comprises a surface element in the focus region.

5. The method of any one of claims 1 to 4, wherein analyzing (410) the second imaging data to determine the status of injection of the fluid into the one borehole comprises applying a machine-learning model to the second imaging data.Docket No.: DRS2993PC006. The method of claim 5, wherein the machine-learning model comprises at least one deep learning model.

7. The method of claim 6, wherein the at least one deep learning model comprises a convolutional neural network, CNN, model.

8. The method of any one of claims 1 to 7, comprising:determining (402) a position and angle of one or more joints of the at least one boom having the drilling and bolting implement coupled thereto; andidentifying (406) the focus region further based on the position and angle of one or more joints of the boom.

9. The method of any one of claims 1 to 8, comprising, as the second imaging data is being analyzed to determine the status of injection of the fluid into the one borehole, dynamically adjusting the focus region based at least on one or more characteristics of an environment in the focus region.

10. The method of any one of claims 1 to 9, wherein determining the status of injection of the fluid comprises detecting fluid outside of the one borehole,the method comprising, in response to the detection of the fluid outside of the one borehole, controlling the fluid injection arrangement to terminate the injection of the fluid into the one borehole.

11. The method of any one of claims 1 to 9, wherein determining the status of the injection of the fluid comprises detecting a leakage in the fluid injection arrangement,the method comprising, in response to the detection of the leakage, controlling the fluid injection arrangement to terminate the injection of fluid into the one borehole and generate an error signal.

12. The method of any one of the preceding claims, wherein controlling (408) the at least one movable imaging device to acquire the second imaging data from the focus region is performed in response to initiating the fluid injection arrangement to begin injecting the fluid into the one borehole.

13. The method of any one of the preceding claims, comprising selecting an imaging device of the at least one movable imaging device to acquire the second imaging data from the focus region.

14. A control device (100, 600) for controlling operation of a mining and / or construction machine for drilling and rock bolting a plurality of boreholes in a rock surface, the mining and / or construction machine comprising at least one boom, a drilling and bolting implement coupled to the at least one boom, a detection system comprising at least one movable imaging device, and a fluid injection arrangement for injecting a fluid into the plurality ofDocket No.: DRS2993PC00boreholes, wherein the control device (100, 600) is configured to perform the method of any one of claims 1 to 13.

15. A mining and / or construction machine (10) for drilling and rock bolting a plurality of boreholes in a rock surface, the mining and / or construction machine comprising at least one boom (12), a drilling and bolting implement (14) coupled to the at least one boom, a detection system (16) comprising at least one movable imaging device (18), a fluid injection arrangement (20) for injecting a fluid into the plurality of boreholes, and a control device (100) of claim 14 for controlling operation of the mining and / or construction machine (10).

16. A computer-program product comprising computer-executable instructions which, when executed by at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 13.

17. A non-transitory tangible computer-readable storage medium comprising computer-executable instructions which, when executed by at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 13.

Images