Support device, method, program, and system

Abstract

Efficiency of farm work is improved. A support device 2 according to one embodiment of the present disclosure includes: an image acquisition unit (imaging unit 20) for acquiring an image of a plant body in a field; a control unit 10 for determining, on the basis of the image, work to be executed on the plant body by a worker during a period in which the plant body is planted in the field; and a marking unit 30 for attaching a mark indicating the determined work.

Description

Support Device, Method, Program, and System

[0001] The present disclosure relates to a support device, method, program, and system.

[0002] Conventionally, technologies for supporting agricultural work are known. For example, Patent Document 1 discloses a plant sorting device for labor-saving sorting of harvested agricultural products.

[0003] Japanese Patent Application Laid-Open No. 2012-191903

[0004] However, with the technology described in Patent Document 1, agricultural work cannot be sufficiently streamlined. For example, there is room for consideration regarding the types of work that can be supported.

[0005] The present disclosure provides a support device, method, program, and system that can streamline agricultural work.

[0006] A support device according to one aspect of the present disclosure includes an image acquisition unit that acquires an image of a plant body in a field, a control unit that determines, based on the image, the work that an operator should perform on the plant body during the period when the plant body is planted in the field, and a marking unit that attaches a mark indicating the determined work.

[0007] A method according to another aspect of the present disclosure includes the support device acquiring an image of a plant body in a field, determining, based on the image, the work that an operator should perform on the plant body during the period when the plant body is planted in the field, and attaching a mark indicating the determined work.

[0008] A program according to another aspect of the present disclosure causes a computer of a support device including a marking unit to acquire an image of a plant body in a field, determine, based on the image, the work that an operator should perform on the plant body during the period when the plant body is planted in the field, and cause the marking unit to attach a mark indicating the determined work.

[0009] A system according to another aspect of the present disclosure performs the following: acquiring images of plants in a field; determining, based on the images, the tasks that a worker should perform on the plants during the period they are planted in the field; and marking the determined tasks.

[0010] According to this disclosure, agricultural work can be made more efficient.

[0011] This is a conceptual diagram relating to the operation of the support device 2 according to this embodiment when marking a plant. This is a conceptual diagram relating to the worker performing work according to the mark placed on the plant. This is a diagram illustrating an example of the functional configuration of the system 1 according to this embodiment. This is a diagram illustrating an example of the operation of the support device 2 according to this embodiment. This is a diagram illustrating an example of the operation of the support device 2 according to this embodiment. This is a diagram illustrating an example of the operation of the support device 2 according to this embodiment. This is a diagram illustrating an example of the operation of the support device 2 according to this embodiment. This is a diagram illustrating an example of the display screen of the terminal device 3 according to this embodiment. This is a diagram illustrating an example of the hardware configuration of each device in the system 1 according to this embodiment.

[0012] <1. Overview> An overview of System 1 according to this embodiment (hereinafter simply referred to as "System 1") will be described with reference to Figures 1A and 1B.

[0013] Figure 1A is a conceptual diagram showing the operation of the support device 2 included in System 1. In one embodiment, the support device 2 comprises an imaging unit 20, a control unit 10, a marking unit 30, and a moving unit 50. The imaging unit 20 acquires images of plants planted in the field by photographing them. Based on the images acquired by the imaging unit 20, the control unit 10 determines the tasks (hereinafter referred to as "target tasks") that workers should perform on the plants during the period they are planted in the field. The marking unit 30 places a mark indicating the tasks determined by the control unit 10, for example, on the plant or in the vicinity of the plant. The support device 2 moves to multiple plants in the field using the moving unit 50 and performs the above series of processes (i.e., photography by the imaging unit 20, determination of tasks by the control unit 10, and marking by the marking unit 30) for each of them.

[0014] In one example, the control unit 10 may determine pruning as the target operation, in which case the marking unit 30 may mark the branches of the plant that are to be pruned. In another example, the control unit 10 may determine pesticide application as the target operation, in which case the marking unit 30 may mark the soil near the roots of the plant.

[0015] As shown, the marks placed on the plants allow workers to easily understand the tasks that need to be performed on them. Figure 1B is a conceptual diagram illustrating how workers perform the tasks corresponding to the marks.

[0016] For example, if a mark indicating pruning is placed on a branch, the worker may prune the branch with that mark. In another example, if a mark indicating pesticide application is placed on the soil near the roots of a plant, the worker may apply pesticide to that plant.

[0017] Here, performing tasks on plants while they are planted in the field sometimes requires advanced judgment skills. One example of a situation requiring advanced judgment skills is when it is necessary to observe the entire plant and decide which branches should be pruned. Another example of a situation requiring advanced judgment skills is when it is necessary to observe the condition of the leaves and roots of the plant and decide whether or not to spray pesticides. However, acquiring such judgment skills requires a lot of experience for workers. Therefore, it is conceivable to have autonomously operating agricultural machinery perform the judgments and tasks. However, with such agricultural machinery, even in tasks such as flower thinning, recognizing and cutting branches connected to fruit would take a very long time for each individual movement. This is because the actuators of typical robots have their operating speed limited for safety reasons, and while tasks such as pruning and harvesting require a certain amount of torque, robots are usually designed to avoid sudden movements. Therefore, even with full automation, the overall work efficiency of the field may actually decrease. This point becomes more pronounced as the field becomes larger. Furthermore, even if autonomously operating agricultural machinery is allowed to make decisions and perform tasks, the quality of its work is often inferior to that of humans, and it cannot perform tasks with sufficient precision.

[0018] The support device 2 according to this embodiment can solve the problems related to such an example. In one embodiment, the support device 2 determines the work to be performed based on an image of the plant and marks the work (see Figure 1A). The worker then performs the work corresponding to the mark while checking the mark (see Figure 1B). In other words, in one embodiment, the support device 2 performs the judgments that require a lot of experience, while the worker performs the actual work that requires delicacy and speed. As a result, even if the worker does not possess advanced judgment abilities, they can perform work based on appropriate judgments with delicacy and speed.

[0019] In the following, a detailed embodiment of System 1 will be described with reference to Figures 2 to 9.

[0020] <2. Functional Configuration> The functional configuration of System 1 will be described with reference to Figure 2. System 1 includes a support device 2, a terminal device 3, an information processing device 4, and a communication network 5. The support device 2, the terminal device 3, and the information processing device 4 are configured to communicate with each other via the communication network 5.

[0021] [Terminal device 3] Terminal device 3 is a communication device used by the person managing the support device 2. The person managing the support device 2 may be, for example, a worker, or the worker's employer. Terminal device 3 may be, for example, a smartphone, a personal computer, a tablet terminal, or a wearable device. Terminal device 3 is equipped with an input interface, an output interface, and a communication interface.

[0022] The input interface is an interface that allows the terminal device 3 to receive input from the user. The input interface may be a touch panel, microphone, camera, keyboard, mouse, etc.

[0023] The output interface is an interface for transmitting information to the user through images, sound, etc. The output interface includes a display (which may also function as a touch panel) and speakers.

[0024] The communication interface is an interface for enabling communication with other devices via the communication network 5. The communication interface may be a wireless communication interface or a wired communication interface.

[0025] The terminal device 3 may, for example, access the services provided by the information processing device 4 via a web browser, or it may access the services by installing dedicated software.

[0026] [Information Processing Device 4] The information processing device 4 acquires information from the support device 2 and transmits display information (in one example, a work plan) generated based on that information to the terminal device 3. In one embodiment, the information processing device 4 is a server device when the support device 2 and the terminal device 3 are client devices. In one embodiment, the information processing device 4 is a cloud server device. The information processing device 4 may be a device that includes, for example, one or more virtual or physical web server devices and one or more virtual or physical database server devices.

[0027] [Support device 2] Support device 2 comprises an imaging unit 20, a marking unit 30, a coordinate acquisition unit 40, a movement unit 50, a control unit 10, a storage unit 12, and a network interface unit 14. These components are electrically connected to each other via a bus 16.

[0028] (Imaging Unit 20) The imaging unit 20 acquires images of plants in the field by photographing them. The imaging unit 20 is, for example, an RGB camera and an infrared camera.

[0029] In one embodiment, the imaging unit 20 photographs the plant according to the shooting parameters. Examples of shooting parameters include aperture, shutter speed, and ISO sensitivity. Other examples of shooting parameters include resolution, image size, and magnification. In one embodiment, the shooting parameters can be changed by the image control function 108 of the control unit 10, which will be described later.

[0030] In one embodiment, the imaging unit 20 is configured so that at least one of its relative position and direction with respect to the support device 2 is adjustable. In one example, the imaging unit 20 may be attached to the tip of a robot arm of the support device 2, and its relative position and direction with respect to the support device 2 may be adjusted by the movement of the robot arm. In another example, the imaging unit 20 may be mounted on a rotatable / slidable base, and its relative position and direction with respect to the support device 2 may be adjusted by the rotation / sliding of the base. In one embodiment, at least one of the relative position and direction of the imaging unit 20 with respect to the support device 2 can be changed by an imaging control function 108 of the control unit 10, which will be described later.

[0031] (Marking section 30) The marking section 30 is used to mark the target work. The mark may be, for example, spray paint, laser engraving, stamping, dot engraving, ribbon, tape, tag (for example, RFID tag, etc.), and LED.

[0032] The physical configuration of the marking section 30 can be appropriately determined depending on the type of mark to be applied. In one example, when the mark is applied by spray, the marking section 30 may be a spray device. In another example, when the mark is applied by laser, the marking section 30 may be a laser irradiation device.

[0033] In one embodiment, the marking unit 30 is configured so that at least one of its relative position and direction with respect to the support device 2 is adjustable. In one example, the marking unit 30 may be attached to the tip of a robot arm of the support device 2, and its relative position and direction with respect to the support device 2 may be adjusted by the movement of the robot arm. In another example, the marking unit 30 may be mounted on a rotatable / slidable base, and its relative position and direction with respect to the support device 2 may be adjusted by the rotation / sliding of the base. In one embodiment, at least one of the relative position and direction of the marking unit 30 with respect to the support device 2 can be changed by a marking control function 106 of the control unit 10, which will be described later.

[0034] In one embodiment, the marking control function 106 controls the marking unit 30 to mark the plant body on which the target operation is to be performed. In this case, the marking control function 106 may control the marking unit 30 to mark the position on the plant body where the target operation is to be performed (hereinafter referred to as the "work position").

[0035] In one embodiment, the marking control function 106 controls the marking unit 30 to mark objects (e.g., soil, shelves, stakes, fences, and poles, etc.) near the plant body on which the target work is to be performed.

[0036] (Coordinate acquisition unit 40) The coordinate acquisition unit 40 acquires the coordinates of the support device 2. These coordinates may be absolute coordinates based on, for example, GPS, or coordinates indicating a section in the field where the support device 2 operates. Alternatively, the coordinates may be relative coordinates based on the positional relationship with reference points in the field or other devices.

[0037] (Movement 50) The movement 50 makes the support device 2 mobile. In one embodiment, the movement 50 is a wheel, making the support device 2 drivable. In one embodiment, the movement 50 is a propeller, making the support device 2 drivable.

[0038] (Control Unit 10) The control unit 10 executes the work content determination function 100, the work position determination function 102, the mark type determination function 104, the marking control function 106, the imaging control function 108, the setting information acquisition function 110, and the recording function 112 by executing various programs stored in the memory unit 12, which will be described later.

[0039] The work content determination function 100 determines the target work based on the image acquired by the imaging unit 20. Determining the target work may include either selecting the target work to be actually performed from one or more candidate target work, or determining (judging) the degree of confidence regarding whether or not the target work should be performed. Here, the degree of confidence may be an indicator of the validity of the judgment regarding whether or not the target work should be performed. The degree of confidence may be determined based on one or more parameters. The degree of confidence may be expressed as a numerical value (e.g., 0 to 100) or as a scale (e.g., low, medium, high).

[0040] In one aspect, the parameter for determining the confidence level may be information regarding the characteristics of the plant body specified or estimated from the image acquired by the imaging unit 20. An example of the parameter at this time may be the density of branches, the number of overlapping leaves, the ratio of cavities in the branches, the distance from the trunk, the age of each branch, the growth momentum, and the age of the tree itself.

[0041] In another aspect, the parameter for determining the confidence level may be various information acquired via a sensor provided in the support device 2 or configured to be communicable with the support device 2. An example of the parameter at this time may be the state of the soil and weather information.

[0042] In another aspect, the parameter for determining the confidence level may be information set by the person who manages the support device 2. An example of the parameter at this time may be the yield of the previous year and the degree of adjustment for the growth of the next year.

[0043] The confidence level may be determined based on a plurality of parameters. According to this configuration, it may be possible to determine the confidence level in consideration of more complex factors. In one example, the work content determination function 100 sets a threshold for each of the two parameters, the number of fruits on the plant and the density of the foliage, and determines a high confidence level when both parameters of the number of fruits and the density of the foliage exceed the threshold (that is, when the branches should be pruned), and determines a low confidence level in other cases (that is, when the branches should not be pruned). According to the configuration of this example, even when the leaves are thick, if the branch has fruits, pruning is necessary because the sunlight is blocked by the thick leaves and as a result the growth of the fruits is inhibited, but if the branch does not have fruits, it is a branch to be used next year and should not be pruned. Such a relatively complex decision-making can be reflected in the processing by the support device 2.

[0044] The work position determination function 102 determines the work position based on the image acquired by the imaging unit 20. Determining the work position may include selecting the work position where the target work should actually be performed from one or more work position candidates.

[0045] The mark type determination function 104 determines the type of mark according to the target work. Determining the type of mark may include, for example, determining the color, shape, size, orientation, and method of the mark (such as whether to mark by spraying or by stamping, etc.).

[0046] The marking control function 106 controls the operation of the marking unit 30. In one embodiment, the marking control function 106 controlling the operation of the marking unit 30 includes controlling the marking unit 30 to apply a mark corresponding to the target work determined by the work content determination function 100. In one embodiment, the marking control function 106 controlling the operation of the marking unit 30 includes controlling the marking unit 30 to apply a mark at a position corresponding to the work position determined by the work positioning function 102. Also, as described above, the marking control function 106 controlling the operation of the marking unit 30 may include changing at least one of the relative position and direction of the marking unit 30 with respect to the support device 2.

[0047] The imaging control function 108 controls the operation of the imaging unit 20. As described above, the imaging control function 108 controlling the operation of the imaging unit 20 may include changing imaging parameters. Also, the imaging control function 108 controlling the operation of the imaging unit 20 may include changing at least one of the relative position and direction of the imaging unit 20 with respect to the support device 2.

[0048] The setting information acquisition function 110 acquires the setting information described later. In one example, the setting information may be input by an operator via, for example, the terminal device 3. In another example, the setting information may be generated by the information processing device 4 and transmitted to the support device 2.

[0049] The recording function 112 records the history information regarding the coordinates where the mark is applied. The recording function 112 may record the history information in the storage unit 12, or may record the history information by transmitting the history information to the information processing device 4.

[0050] (Storage Unit 12) The storage unit 12 stores various information necessary for the operation of the support device 2. In one embodiment, the storage unit 12 stores the program to be executed by the control unit 10.

[0051] (Network Interface Unit 14) The network interface unit 14 enables communication with other devices via the communication network 5.

[0052] These functions will be described in more detail in the following embodiments.

[0053] <3. Operation> The following describes various configurations that can be adopted in the support device 2 according to the first to sixth embodiments. Those skilled in the art can configure the support device 2 by appropriately combining the items described in the following first to sixth embodiments.

[0054] [First Embodiment] The operation of the support device 2 according to the first embodiment will be described with reference to Figures 3 and 4. Figure 3 is a flowchart showing an example of the operation of the support device 2.

[0055] First, the imaging unit 20 acquires an image of a plant in the field (S100). In the example shown in Figure 3, the support device 2 acquires an image of an entire plant.

[0056] Next, the work content determination function 100 determines the target work based on the image acquired in step S100 (S102). In the example in Figure 3, the support device 2 decides to perform pruning based on the acquired image of the entire plant. Here, pruning is an example of a "target work".

[0057] Next, the work position determination function 102 determines the work position (S104) based on the image acquired in step S100 and the target work determined in step S102. In the example in Figure 3, the support device 2 determines a specific position on a specific branch of a plant in order to perform pruning. Here, the specific position on a specific branch is an example of a "work position".

[0058] Next, the mark type determination function 104 determines the type of mark corresponding to the target work determined in step S102 (S106). In the example in Figure 3, the support device 2 determines a red spray as the type of mark corresponding to pruning. Here, the red spray is just one example of a "type of mark".

[0059] Next, the marking control function 106 controls the marking unit 30 to apply a mark of the type determined in step S106 to a position corresponding to the work position determined in step S104 (S108). In the example in Figure 3, red spray paint is applied to a specific position on a specific branch. Here, the specific position on a specific branch is an example of a "position corresponding to the work position," and applying red spray paint is an example of "applying a mark of the determined type."

[0060] Next, the support device 2 determines whether or not to mark other plants (S110). If the result in step S110 is YES, the support device 2 moves to the other plant using the moving unit 50 (S112), acquires an image of the other plant (S100), and performs the subsequent processing. If the result in step S110 is NO, the support device 2 terminates its operation.

[0061] Figure 4 is a diagram illustrating the operation of the support device 2, which was described with reference to Figure 3, from a different perspective. Steps S100 to S108 in Figure 4 correspond to steps S100 to S108 in Figure 3.

[0062] After the imaging unit 20 acquires an image (S100), the work content determination function 100 determines the target work based on the image (S102). Figure 3 illustrates an example where pruning is the target work, but the target work could be, for example, bud removal, cutting, and pesticide spraying.

[0063] For example, if the acquired image identifies that the area and / or number of leaves on a plant body are greater than a preset threshold, the work content determination function 100 may determine that pruning is the target work.

[0064] In another example, if the acquired image identifies that the number of buds on a plant exceeds a predetermined threshold, the work content determination function 100 may determine that bud thinning is the target work. The predetermined threshold can be set based, for example, on the number of fruits that should develop per bud or per plant.

[0065] In another example, if, based on the acquired images, it is identified that the plant has characteristics such as the presence of pests or diseases, a large number of black spots on the leaves, or a large number of insects (which may be observed as white dots in the images), the work content determination function 100 may determine that cutting is the target work.

[0066] In another example, if pests or diseases are detected on a plant based on the acquired images, the work content determination function 100 may determine that pesticide spraying is the target work.

[0067] In another example, if the acquired image determines that no target operation needs to be performed on the plant, the operation content determination function 100 decides not to determine a target operation or not to perform the target operation. In this case, the support device 2 does not need to perform any further processing.

[0068] If the work content determination function 100 decides to perform the target work, the work position determination function 102 determines the work position based on the image and the target work (S104).

[0069] For example, as explained with reference to Figure 3, if the target task is pruning, the work location may be a specific position on a specific branch. In this case, the work location determination function 102 can determine which of the multiple branches of the plant is the "specific branch" and which part of that specific branch is the "specific position" based on an image of the plant.

[0070] For example, if the task is pruning, the work location could be a specific bud (lateral bud). In this case, the work location determination function 102 can determine which of the multiple buds growing on the plant is the "specific bud" based on an image of the plant.

[0071] In another example, if the task is bud removal, the work location could be a specific bud. In this case, the work location determination function 102 can determine which of the multiple buds on the plant is the "specific bud" based on an image of the plant.

[0072] In another example, if the target operation is cutting, the work location may be a specific leaf. In this case, the work location determination function 102 can determine which of the multiple leaves of the plant is the "specific leaf" based on an image of the plant.

[0073] In another example, if the task involves spraying pesticides, the work area could be the entire plant.

[0074] Next, the mark type determination function 104 determines the type of mark according to the target operation (S106). Figure 3 illustrates an example in which a red spray mark is determined as the type of mark corresponding to pruning, but in one example, the type of mark corresponding to bud removal may be a blue spray, the type of mark corresponding to cutting may be a yellow spray, and the type of mark corresponding to pesticide spraying may be a black spray.

[0075] Next, the marking control function 106 determines the position to affix the mark (S108). The mark is affixed to a position corresponding to the work position. In one embodiment, the "position corresponding to the work position" may be a position common to the work position (for example, the work position itself or a part thereof), a position in the vicinity of the work position, and a position from which the worker can substantially identify the work position.

[0076] In one example, as explained with reference to Figure 3, when the work position is a specific position on a specific branch, the marking control function 106 decides to place a mark at that specific position. In this case, the work position and the position to be marked are the same.

[0077] In another example, the marking control function 106 determines that if the work location is a specific bud, the support device 2 will mark the stem on which that specific bud is growing. In this case, the work location and the location where the mark is placed are not the same, but the worker can substantially identify the work location by the location where the mark is placed.

[0078] In another example, the marking control function 106 determines that if the work location is a specific leaf, the support device 2 will mark that specific leaf. In this case, the work location and the location where the mark is to be placed are the same.

[0079] In another example, the marking control function 106 determines that if the work area is the entire plant, the support device 2 will place a mark on the ground near the plant. In this case, the work area and the location where the mark is placed are not the same, but the worker can substantially identify the work area by the location where the mark is placed.

[0080] As described above, the support device 2 comprises an imaging unit 20 that acquires images of plants in the field, a control unit 10 (work content determination function 100) that determines, based on the images, the work that the worker should perform on the plant during the period that the plant is planted in the field, and a marking unit 30 that marks the determined work. This makes it easy for the worker to understand what work needs to be done on the plant.

[0081] In one embodiment, the control unit 10 determines the work location on the plant where the determined task should be performed (work location determination function 102), and controls the marking unit 30 to place a mark at the location corresponding to the work location (marking control function 106). With this configuration, the worker can easily grasp not only that a task should be performed but also the work location.

[0082] In one embodiment, the control unit 10 determines the type of mark based on the determined task (mark type determination function 104). With this configuration, the worker can easily understand not only that a task should be performed, but also the content of the task to be performed.

[0083] In one embodiment, the support device 2 includes a mobile unit 50 that moves from one plant to another. With this configuration, the support device 2 can autonomously mark multiple plants in the field.

[0084] The support device 2 can periodically (for example, daily) patrol the field according to steps S100 to S112 and transmit various information acquired and determined (for example, images of plants, target work, work location, type of mark determined based on the images, etc.) to the information processing device 4. As a result, the information processing device 4 can generate information indicating where, when, and what kind of work should be performed (or has been performed) in the field.

[0085] [Second Embodiment] In the second embodiment, an example is described in which the mark type determination function 104 determines the type of mark based on setting information. The setting information is also information for interpreting the meaning of the mark that is applied, and includes, for example, information on the content of the work for each mark, the time when the mark is applied, and at least one of the location on the plant body to which the mark is applied. The setting information can be set via the terminal device 3.

[0086] Figure 5 shows an example of the configuration information. In the configuration information example in Figure 5, the types of marks "red spray" and "black spray" are associated with the candidate target tasks, "pruning" and "pesticide spraying," respectively.

[0087] In one example, if the work content determination function 100 determines "pruning" as the target work, the mark type determination function 104 will determine "red spray" associated with "pruning" in the setting information as the mark type. In another example, if the work content determination function 100 determines "pesticide spraying" as the target work, the mark type determination function 104 will determine "black spray" associated with "pesticide spraying" in the setting information as the mark type. Note that "pruning" and "pesticide spraying" are examples of target work options.

[0088] Figure 6 shows another example of the setting information. In the example setting information in Figure 6, the type of mark, "triangle" for "red spray," is associated with the combination of the time point to mark "time point t1" and the candidate target work, "pruning." Also, the type of mark, "triangle" for "black spray," is associated with the combination of the time point to mark "time point t1" and the candidate target work, "pesticide spraying." Also, the type of mark, "circle" for "red spray," is associated with the combination of the time point to mark "time point t2" and the candidate target work, "pruning." Also, the type of mark, "circle" for "black spray," is associated with the combination of the time point to mark "time point t2" and the candidate target work, "pesticide spraying."

[0089] In one example, if the work content determination function 100 determines "pruning" as the target work and the time when the marking unit 30 makes a mark is "time t1", the mark type determination function 104 determines the mark type to be a "triangle" for "red spray". In another example, if the work content determination function 100 determines "pesticide spraying" as the target work and the time when the marking unit 30 makes a mark is "time t2", the mark type determination function 104 determines the mark type to be a "circle" for "black spray".

[0090] The support device 2 can, for example, mark the same plant multiple times if the field is inspected multiple times at different points in time. With this configuration, the worker can easily understand when each of the multiple marks placed on the plant was made, and as a result, can understand, for example, which mark is the most recent.

[0091] Figure 7 shows another example of the setting information. In the example setting information in Figure 7, the combination of the location of the plant marked as "Northwest section of the field" and the candidate target operation "Pruning" is associated with the type of mark "Triangle" with "Red spray". Also, the combination of the location of the plant marked as "Northwest section of the field" and the candidate target operation "Pesticide spraying" is associated with the type of mark "Triangle" with "Black spray". Also, the combination of the location of the plant marked as "Southeast section of the field" and the candidate target operation "Pruning" is associated with the type of mark "Circle" with "Red spray". Also, the combination of the location of the plant marked as "Southeast section of the field" and the candidate target operation "Pesticide spraying" is associated with the type of mark "Circle" with "Black spray".

[0092] In one example, if the work content determination function 100 determines "pruning" as the target work, and the location of the plant to be marked by the marking unit 30 (≒ coordinates of the support device 2 acquired by the coordinate acquisition unit 40) is the "northwest section of the field," then the mark type determination function 104 determines the mark type to be a "triangle" with "red spray paint." In another example, if the work content determination function 100 determines "pesticide spraying" as the target work, and the location of the plant to be marked by the marking unit 30 is the "southeast section of the field," then the mark type determination function 104 determines the mark type to be a "circle" with "black spray paint."

[0093] In field work, tasks are not always performed on the entire field at once; sometimes the field is divided into sections, and work is carried out in each section. With the above configuration, workers can easily understand where the marked plants are located in the field (i.e., where they are currently working). As a result, it is possible to prevent, for example, workers from mistakenly working in a section they did not intend to work in.

[0094] In one embodiment, the imaging control function 108 controls the acquisition of images by the imaging unit 20 based on the setting information.

[0095] For example, if "pruning" is included in the list of target tasks specified in the configuration information, the imaging control function 108 controls the imaging parameters of the imaging unit 20, its relative position to the support device 2, and its orientation so as to capture the entire plant. This is because it is necessary to observe the entire plant in order to determine whether pruning is necessary and where it should be done.

[0096] In another example, if "pesticide spraying" is included among the candidate target operations specified in the configuration information, the imaging control function 108 controls the imaging parameters of the imaging unit 20, as well as its relative position and direction to the support device 2, to photograph a specific part of the plant (for example, leaves near the ground). This is because, in order to determine whether or not pesticide spraying is necessary, it is necessary to pay particular attention to the areas where symptoms of pests and diseases appear.

[0097] In another example, if the candidate target operations specified in the configuration information include both "pruning" and "pesticide spraying," the imaging control function 108 controls the imaging parameters of the imaging unit 20, as well as its relative position and orientation to the support device 2, so as to capture images of the entire plant and specific parts of the plant, respectively.

[0098] This configuration allows for the acquisition of appropriate images according to the purpose of operation of the support device 2 as defined by the configuration information.

[0099] [Third Embodiment] In one embodiment, the imaging unit 20 acquires a first image of the plant at a first time point and a second image of the same plant at a second time point different from the first time point, and the work content determination function 100 determines the target work based on the first image and the second image. In one embodiment, the work content determination function 100 determines the target work based on the state of the plant identified based on the first image and the change in the state of the plant identified based on the second image.

[0100] For example, if the leaves of the plant identified based on the first image are in a normal state (for example, with very few black spots and very few insects), and the leaves of the plant identified based on the second image have many black spots or insects, then it is determined that a pest or disease occurred between the first and second time points, and the work content determination function 100 determines that pesticide spraying is the target work.

[0101] In another example, if the fruit of the plant identified based on the first image is green, and the fruit of the plant identified based on the second image is reddish, the work content determination function 100 will determine that harvesting the fruit is the target work.

[0102] With this configuration, the support device 2 will be able to determine the target operation while taking into account the changes in the plant body over time.

[0103] [Fourth Embodiment] In one embodiment, the work content determination function 100 can determine the confidence level of the target work when determining the target work based on an image of the plant. In one example, if the work content determination function 100 determines that pruning is clearly necessary for the plant, pruning is determined as the target work, and a relatively high confidence level is determined. In another example, if the work content determination function 100 determines that pruning is not clearly necessary for the plant, pruning is determined as the target work, and a relatively low confidence level is determined. Whether or not pruning is clearly necessary for the plant can be determined, for example, based on whether the density of the branches and the number of overlapping leaves of the plant are excessively high.

[0104] In one embodiment, the marking control function 106 controls the marking unit 30 to mark according to at least one of the first mode and a second mode different from the first mode, based on the confidence level. In one example, if the confidence level is above a predetermined threshold, the marking control function 106 controls the marking unit 30 to mark a "circle" mark (an example of a mark in the first mode), and if the confidence level is below the predetermined threshold, the marking control function 106 controls the marking unit 30 to mark a "triangle" mark (an example of a mark in the second mode).

[0105] With this configuration, the worker can decide whether to perform (or not perform) the target task after understanding the confidence level of the support device 2. This makes it easier for the worker to make a decision on whether or not to perform the target task, while leaving the final decision to the worker, and as a result, the work may be sped up. Furthermore, with the above configuration, it may be easier for those in charge of plant cultivation to give instructions to workers. For example, those in charge of plant cultivation can instruct workers to prune branches marked with a "circle" and to seek the judgment of an expert for branches marked with a "triangle". In another example, those in charge of plant cultivation can instruct workers to prune branches marked with a blue mark, to have an expert judge branches marked with a yellow mark, and not to prune branches marked with a red mark. This may lead to the standardization of work procedures.

[0106] [Fifth Embodiment] In one embodiment, the recording function 112 acquires information regarding the marks made by the marking unit 30 and information regarding the coordinates of the support device 2 when the marking unit 30 made the marks, and records this information as history information by associating it with other information.

[0107] In one embodiment, each plant is assigned an ID, and the recording function 112 may identify the plant ID from the coordinate information and record history information by associating the mark information with the plant ID, or it may identify a section within the field from the coordinate information and record history information by associating the mark information with the section.

[0108] In one embodiment, the information processing device 4 may generate a work plan for tasks to be performed on plants in the field based on the received history information. The work plan may include information relating the number and / or type of marks placed in each of the multiple sections of the field (i.e., information indicating where and what tasks should be performed). The work plan may also include information regarding tasks to be performed on plants in the field at each of one or more point in time (i.e., information indicating when and what tasks should be performed). The information processing device 4 may then transmit the work plan to the terminal device 3.

[0109] Figure 8 is a diagram illustrating an example screen of terminal device 3 that displays a work plan. The example screen in Figure 8 displays a field map d10. The field map d10 is divided into sections 1 to 9. Sections 1 to 9 can each correspond to actual field divisions.

[0110] According to field map d10, section 1 has three red marks and one blue mark. Section 2 has one red mark and one blue mark. Section 3 has three red marks and two blue marks. Section 4 has two red marks and two blue marks. Section 5 has two red marks and one blue mark. Section 6 has two red marks and three blue marks. Section 7 has four blue marks. Section 8 has one red mark and three blue marks. Section 9 has two red marks and two blue marks. By checking field map d10, workers can get an overview of which section of the field should be used and what kind of work should be performed.

[0111] [Sixth Embodiment] In one embodiment, the control unit 10 acquires information about other marks previously applied to the plant based on an image of the plant, and the marking control function 106 controls the marking unit 30 to apply a mark based on the information about the other marks.

[0112] For example, if the position corresponding to the work position determined by the work position determination function 102 (i.e., the position where the mark should be placed) overlaps with a position where another mark has already been placed, the marking control function 106 can control the marking unit 30 to place the mark in a position different from the other marks.

[0113] For example, the work content determination function 100 may determine the next target work to be performed by taking into account information about other marks previously attached to the plant body, in addition to the image, i.e., the target work that has been performed in the past.

[0114] In one example, the recording function 112 may detect whether the work has been performed based on information and images of other marks previously attached to the plant, and record whether the work has been performed or not in the history information.

[0115] With this configuration, the support device 2 can add further marks later, taking into account the positions of other marks that have been added previously.

[0116] <4. Hardware Configuration> Referring to Figure 9, an example of a hardware configuration when the devices included in System 1 described above are implemented by the computer 70 will be explained. Note that the functions of each device can also be implemented by dividing them among multiple devices.

[0117] As shown in Figure 9, the computer 70 includes a processor 700, a storage device 702, an input interface 704, a data interface 706, a communication interface 708, and a display device 710.

[0118] The processor 700 controls various processes in the computer 70 by executing programs stored in the storage device 702. For example, each functional unit of the control unit 10 of the support device 2 can be realized by the processor 700 executing programs stored in the storage device 702.

[0119] The storage device 702 is a storage medium such as RAM (Random Access Memory). RAM temporarily stores the program code of the program executed by the processor 700, as well as data required when the program is executed.

[0120] The storage device 702 can also be a non-volatile storage medium such as a hard disk drive (HDD) or flash memory. The storage device 702 stores the operating system and various programs for realizing the above configurations. The storage medium storing these various programs may be a computer-readable non-temporary storage medium (non-transitor computer-readable medium). In addition, the storage device 702 can also store tables for registering various information and a database (DB) for managing these tables. Such programs and data are loaded into the storage device 702 as needed and accessed by the processor 700.

[0121] The input interface 704 is a device for receiving input from the user. Specific examples of the input interface 704 include cameras, buttons, microphones, keyboards, mice, touch panels, various sensors, and wearable devices. The input interface 704 may be connected to the computer 70 via an interface such as USB (Universal Serial Bus).

[0122] The data interface 706 is a device for inputting data from outside the computer 70. Specific examples of the data interface 706 include drive devices for reading data stored on various storage media. The data interface 706 may also be located outside the computer 70. In that case, the data interface 706 would be connected to the computer 70 via an interface such as USB.

[0123] The communication interface 708 is a device for performing data communication with external devices of the computer 70 via a communication network 5, either by wire or wireless connection. The communication interface 708 may also be located outside the computer 70. In that case, the communication interface 708 is connected to the computer 70 via an interface such as USB.

[0124] The display device 710 is a device for displaying various types of information. Specific examples of the display device 710 include liquid crystal displays, organic EL (Electro-Luminescence) displays, and displays for wearable devices. The display device 710 may be located outside the computer 70. In that case, the display device 710 is connected to the computer 70 via, for example, a display cable. Furthermore, if a touch panel is used as the input I / F 704, the display device 710 can be configured as an integrated unit with the input I / F 704.

[0125] Furthermore, the components of the device included in the system 1 described above are such that a program stored in the storage device 702 is executed by the processor 700, thereby realizing a defined process in cooperation with other hardware. In other words, these components can be envisioned as software or firmware, or as corresponding hardware, and in both concepts, they can be described and interpreted as "function," "means," "part," "processing circuit," "unit," or "module," etc.

[0126] <5. Modifications> The embodiments described above are provided to facilitate understanding of the Disclosure and are not intended to limit the Disclosure. The configurations that the embodiments may have are not limited to those exemplified and can be modified as appropriate. Furthermore, it is possible to partially substitute or combine configurations shown in different embodiments.

[0127] In the above embodiment, pruning, cutting, bud removal, and pesticide application were given as examples of target tasks, but the target tasks are not limited to these. The target tasks may be any tasks that should be performed to manage and cultivate the plant.

[0128] In the above embodiment, the imaging unit 20 was described as being included in the support device 2, but it is not limited to this. The imaging unit 20 may be a separate device from the support device 2. In one example, the imaging unit 20 is a fixed-point camera installed in the field. In another example, the imaging unit 20 is a camera included in the terminal device 3.

[0129] In the above embodiment, the support device 2 was described as having a movable part 50, but it is not limited to this. The support device 2 may not have a movable part 50 and may be carried by, for example, a worker.

[0130] The work content determination function 100 can determine information regarding when and how the target work should be performed when determining the target work. The mark type determination function 104 can determine the type of mark based on this information. With this configuration, a worker who sees the mark will be able to understand when and how the target work should be performed.

[0131] In the above embodiment, the work content determination function 100 was described as determining the target work based on the image acquired by the imaging unit 20, but it is not limited to this. The work content determination function 100 may determine the work content based on setting information stored in the storage unit 12, for example. For example, if the setting information includes information that a mark corresponding to "pruning" should be applied, the work content determination function 100 may determine "pruning" as the target work (in one example, without referring to the image acquired by the imaging unit 20). In this case, the support device 2 may, after determining the target work by referring to the setting information (corresponding to S102 in Figure 3), acquire an image of the plants in the field (corresponding to S100 in Figure 3), and determine the work position based on the image and the target work (corresponding to S104 in Figure 3).

[0132] The system configuration (including the physical configuration) described in the above embodiment is merely an example and can be modified as appropriate based on the knowledge of those skilled in the art, to the extent that it does not create inconsistencies.

[0133] <6. Supplementary Information> The wording in this embodiment may be understood as follows, to the extent that it does not cause any contradiction.

[0134] In this embodiment, "performing a predetermined process based on predetermined information" may mean performing the predetermined process based on at least a portion of the predetermined information, performing the predetermined process based on at least the predetermined information, or performing the predetermined process probabilistically based on the predetermined information. In other words, "performing a predetermined process based on predetermined information" is not limited to performing the predetermined process based solely on the predetermined information.

[0135] In this embodiment, "executing another process based on a predetermined process" may mean any of the following: executing the other process after the predetermined process has been executed; executing the predetermined process and the other process consecutively; executing the other process based on information determined by the predetermined process; executing the other process on the condition that the predetermined process has been executed; or executing the other process by means of the predetermined process. Furthermore, "executing another process by a predetermined process" may be understood in the same way as "executing another process based on a predetermined process."

[0136] In this embodiment, "the predetermined information includes other information" may mean either that at least a portion of the predetermined information is the other information, or that the other information can be obtained based on the predetermined information.

[0137] In this embodiment, "a predetermined process includes other processes" may mean either that at least a part of the predetermined process is the other process (i.e., the other process is performed in the process of obtaining the result of the predetermined process), or that one aspect of the predetermined process is the other process.

[0138] In this embodiment, "a predetermined object and another object correspond" may mean that there is a one-to-one relationship between the predetermined object and the other object, that the other object is included in a predetermined set identified based on the predetermined object, or that the other object can be identified based on the predetermined object. Furthermore, "a predetermined object and another object corresponding" is not limited to being managed, for example, in a database. Also, "a predetermined object and another object being associated" may be understood in the same way as "a predetermined object and another object corresponding."

[0139] In this embodiment, "acquiring information" includes making the information processable by the control unit 10. "Acquiring information" may include, for example, receiving the information from another device, obtaining the information through predetermined processing, and reading the information from the storage unit 12.

[0140] In this embodiment, "generating information" may mean either making the information obtained by a predetermined process processable in the control unit 10, or storing the information obtained by the predetermined process in the storage unit 12.

[0141] In this embodiment, "determining information" may mean either selecting at least one piece of information from one or more pieces of information, or generating new information.

[0142] In this embodiment, "outputting information" may mean either transmitting the information to another device or outputting the information as audio or video.

[0143] <7. Example Configuration> This disclosure includes the following technologies.

[0144] [Note 1] The support device 2 comprises an image acquisition unit (imaging unit 20) for acquiring images of plants in the field, a control unit 10 for determining the work that the worker should perform on the plants during the period that the plants are planted in the field based on the images, and a marking unit 30 for marking the determined work.

[0145] [Note 2] The support device 2 described in Note 1, wherein the control unit 10 determines the work position on the plant body where the determined work should be performed and controls the marking unit 30 to place a mark at the work position.

[0146] [Note 3] The control unit 10 determines the type of mark based on the determined work, and is the support device 2 as described in Note 1 or 2.

[0147] [Note 4] The support device 2 described in Note 3, wherein the control unit 10 acquires setting information relating to at least one of the content of the work, the time at which the mark is applied, and the location of the plant, and further determines the type of mark based on said setting information.

[0148] [Note 5] The control unit 10 controls the acquisition of images by the image acquisition unit (imaging unit 20) based on the setting information, and is the support device 2 described in Note 4.

[0149] [Note 6] The support device 2 according to any one of Notes 1 to 5, wherein the image acquisition unit (imaging unit 20) acquires a first image of the plant at a first time point and a second image of the plant at a second time point different from the first time point, and the control unit 10 determines, based on the first and second images, the work that the worker should perform on the plant during the period that the plant is planted in the field.

[0150] [Note 7] The support device 2 described in Note 6, wherein the control unit 10 determines the work that the worker should perform on the plant during the period the plant is planted in the field, based on the state of the plant identified based on the first image and the change in the state of the plant identified based on the second image.

[0151] [Note 8] The support device 2 according to any one of Notes 1 to 7, wherein the control unit 10 determines the degree of confidence in the work that an operator should perform on the plant during the period that the plant is planted in the field, and controls the marking unit 30 to mark the plant according to at least one of the first mode and a second mode different from the first mode based on the degree of confidence.

[0152] [Note 9] The support device 2 described in any one of Notes 1 to 8 further includes a control unit 10 which acquires information regarding the marks made by the marking unit 30 and information regarding the coordinates of the support device 2 at the time the marking unit 30 made the marks, and records these information in association as history information.

[0153] [Note 10] The support device 2 according to any one of Notes 1 to 9, wherein the control unit 10 acquires information about other marks previously applied to the plant body based on the image, and controls the marking unit 30 to apply further marks based on the information about the other marks.

[0154] [Note 11] The support device 2 described in any one of Notes 1 to 10, wherein the mark is selected from the group which includes at least one of the following: a cutting mark corresponding to the operation of cutting a part of a plant, a pruning mark corresponding to the operation of trimming a part of a plant, and a bud removal mark corresponding to the operation of removing buds from a plant.

[0155] [Note 12] The support device 2 according to any one of Notes 1 to 11, comprising a moving unit 50 that moves from one plant to another.

[0156] [Note 13] A method in which the support device 2 acquires images of plants in the field, determines, based on the images, the work that the worker should perform on the plants during the period that the plants are planted in the field, and marks the determined work.

[0157] [Note 14] A program that causes the computer 70 of the support device 2 equipped with a marking unit 30 to acquire images of plants in the field, determine, based on the images, the work that the worker should perform on the plants during the period that the plants are planted in the field, and affix a mark indicating the determined work using the marking unit 30.

[0158] [Note 15] System 1 performs the following actions: acquiring images of plants in a field, determining, based on the images, the tasks that workers should perform on the plants during the period they are planted in the field, and marking the plants to indicate the determined tasks.

[0159] [Appendix 16] The system described in Appendix 15, further comprising: acquiring historical information that associates the determined work with information regarding the coordinates marked to indicate the work; generating a work plan for the work to be performed on plants in the field based on the historical information; and outputting the work plan to the worker's terminal device.

[0160] 1...System, 2...Support device, 3...Terminal device, 4...Information processing device, 10...Control unit, 12...Storage unit, 20...Imaging unit, 30...Marking unit, 40...Coordinate acquisition unit, 50...Movement unit, 70...Computer, 100...Work content determination function, 102...Work position determination function, 104...Mark type determination function, 106...Marking control function, 108...Imaging control function, 110...Setting information acquisition function, 112...Recording function, 700...Processor

Claims

1. A support device comprising: an image acquisition unit for acquiring images of plants in a field; a control unit for determining, based on the images, the tasks that an operator should perform on the plants during the period that the plants are planted in the field; and a marking unit for marking the determined tasks.

2. The support device according to claim 1, wherein the control unit determines a work position on the plant body where the determined work should be performed, and controls the marking unit to affix the mark to the position corresponding to the work position.

3. The support device according to claim 1, wherein the control unit determines the type of mark based on the determined operation.

4. The support device according to claim 3, wherein the control unit acquires setting information relating to at least one of the content of the work, the time at which the mark is applied, and the location of the plant, and further determines the type of mark based on the setting information.

5. The support device according to claim 4, wherein the control unit controls the acquisition of images by the image acquisition unit based on the setting information.

6. The support device according to claim 1, wherein the image acquisition unit acquires a first image of the plant at a first time point and a second image of the plant at a second time point different from the first time point, and the control unit determines, based on the first image and the second image, the work that the worker should perform on the plant during the period that the plant is planted in the field.

7. The support device according to claim 6, wherein the control unit determines, based on the state of the plant identified based on the first image and the change in the state of the plant identified based on the second image, the work that the worker should perform on the plant during the period that the plant is planted in the field.

8. The support device according to claim 1, wherein the control unit determines the degree of confidence in the work that the worker should perform on the plant during the period that the plant is planted in the field, and controls the marking unit to affix the mark according to at least one of the first mode and a second mode different from the first mode, based on the degree of confidence.

9. The support device according to claim 1, wherein the control unit further acquires information relating to the mark made by the marking unit and information relating to the coordinates of the support device at the time the marking unit made the mark, and records these pieces of information together as historical information.

10. The support device according to claim 1, wherein the control unit acquires information about other marks previously applied to the plant body based on the image, and controls the marking unit to apply the mark based on the information about the other marks.

11. The support device according to claim 1, wherein the mark is selected from the group including at least one of: a cutting mark corresponding to the operation of cutting a part of the plant; a pruning mark corresponding to the operation of trimming a part of the plant; and a bud removal mark corresponding to the operation of removing buds from the plant.

12. The support device according to claim 1, further comprising a moving unit that moves from one plant to another.

13. A method comprising: a support device acquiring images of plants in a field; determining, based on the images, the tasks that a worker should perform on the plants during the period they are planted in the field; and marking the determined tasks.

14. A program that causes a support device equipped with a marking unit to perform the following actions: acquire images of plants in a field; determine, based on the images, the tasks that an operator should perform on the plants during the period that the plants are planted in the field; and affix a mark indicating the determined task using the marking unit.

15. A system that performs the following: acquiring images of plants in a field; determining, based on the images, the tasks that a worker should perform on the plants during the period they are planted in the field; and marking the plants to indicate the determined tasks.

16. The system according to claim 15, further comprising: obtaining historical information relating the determined work to information relating to coordinates marked to indicate the work; generating a work plan for work to be performed on plants in the field based on the historical information; and outputting the work plan to the worker's terminal device.

Images