Method and apparatus for correcting distortion of image by means of selection of structural marker within ship environment

Abstract

The present disclosure relates to a method and apparatus for correcting distortion of an image by means of selection of a structural marker within a ship environment. According to an embodiment, the method for correcting distortion of an image by means of selection of a structural marker within a ship environment may comprise the steps of: selecting a structural marker from among a plurality of structures installed in a ship environment; acquiring a three-dimensional position of the structural marker and the current two-dimensional position of the structural marker; and generating at least one corrected image by correcting distortion included in at least one initial image obtained by photographing the structural marker by using the three-dimensional position and the current two-dimensional position.

Description

Method and apparatus for correcting image distortion through the selection of structural markers within a ship environment

[0001] The present disclosure relates to a method and apparatus for correcting image distortion through the selection of structural markers within a ship environment.

[0002] Recently, due to ship automation, the number of crew members on board is decreasing while the size of ships is increasing.

[0003] This leads to the problem that it becomes difficult to respond quickly to various accidents occurring on ships. For example, fires on container ships cause $20 billion in damages annually.

[0004] Conventional ship control systems had the disadvantage of being unable to monitor information related to ship operations, such as crew members and operational status, in real time. For example, when monitoring using imaging devices, even if an event was detected, it was difficult to immediately pinpoint the area where the event occurred and to devise immediate response measures. However, since the majority of ship accidents occur at sea, a delayed response can lead to even greater damage.

[0005] Accordingly, there is a need for research and development of intelligent control solutions that can be used in various marine environments, such as ships, yards, and shipyards.

[0006] The aforementioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the process of deriving the present invention, and it cannot be considered as prior art disclosed to the general public prior to the filing of the present invention.

[0007] The present disclosure is intended to provide a method and apparatus for correcting image distortion through the selection of structural markers within a ship environment. The problems to be solved by the present invention are not limited to those mentioned above, and other problems and advantages of the present invention not mentioned can be understood from the following description and will be more clearly understood by the embodiments of the present invention. Furthermore, it will be understood that the problems and advantages to be solved by the present invention can be realized by the means and combinations thereof set forth in the claims.

[0008] As a technical means for achieving the technical problem described above, the first aspect of the present disclosure is a method for correcting image distortion by selecting a structure marker in a ship environment, comprising: a step of selecting a structure marker among a plurality of structures installed in a ship environment; a step of obtaining a three-dimensional position of the structure marker and a current two-dimensional position of the structure marker; and a step of correcting distortion included in at least one initial image of the structure marker using the three-dimensional position and the current two-dimensional position to generate at least one corrected image.

[0009] A second aspect of the present disclosure is a device for correcting image distortion by selecting a structure marker in a ship environment, comprising at least one memory and at least one processor, wherein the at least one processor selects a structure marker using movement data for each of a plurality of structures installed in a ship environment, and corrects distortion included in the at least one initial image using the three-dimensional position of the structure marker and the current two-dimensional position of the structure marker to generate at least one corrected image.

[0010] A third aspect of the present disclosure may provide a computer-readable recording medium having a program for executing a method according to a first aspect on a computer.

[0011] In addition to this, other methods for implementing the present invention, other systems, and computer-readable recording media storing a computer program for executing said methods may be further provided.

[0012] Other aspects, features, and advantages other than those described above will become clear from the following drawings, claims, and detailed description of the invention.

[0013] According to the means for solving the problem of the present disclosure described above, distortion in an image can be effectively corrected through a structure suitable for use as a marker among a plurality of structures located within a ship environment.

[0014] In addition, the present disclosure allows for the detection of the occurrence of objects or events based on distortion-corrected images, thereby enabling flexible response to accidents occurring in a ship environment and preventing the occurrence of accidents.

[0015] The effects of the embodiments are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description of the present invention.

[0016] FIG. 1 is a conceptual diagram illustrating a method for generating a control solution using a control device in a ship environment according to one embodiment.

[0017] FIG. 2 is a block diagram of a control device according to one embodiment.

[0018] FIG. 3 is an exemplary configuration diagram of a system including a control device according to one embodiment.

[0019] FIG. 4 is a flowchart illustrating a method for correcting image distortion through the selection of a structure marker within a ship environment according to one embodiment.

[0020] FIG. 5 is an exemplary drawing for explaining a method for correcting distortion in an image of a ship environment according to one embodiment.

[0021] FIG. 6 is an exemplary drawing for explaining a method of selecting a structure marker according to one embodiment.

[0022] FIG. 7 is a flowchart illustrating a method for selecting a structure marker using time-based movement data of structures according to one embodiment.

[0023] FIG. 8 is an exemplary drawing for explaining a method of generating a correction image using the current two-dimensional position of a structure marker according to one embodiment.

[0024] The present disclosure relates to a method and apparatus for correcting image distortion by selecting a structure marker within a ship environment. According to one embodiment, a method for correcting image distortion by selecting a structure marker within a ship environment may include: a step of selecting a structure marker among a plurality of structures installed in a ship environment; a step of obtaining a three-dimensional position of the structure marker and a current two-dimensional position of the structure marker; and a step of generating at least one corrected image by correcting distortion included in at least one initial image captured of the structure marker using the three-dimensional position and the current two-dimensional position. According to another embodiment, an apparatus for correcting image distortion by selecting a structure marker within a ship environment may include at least one memory and at least one processor, wherein the at least one processor may select a structure marker using movement data for each of a plurality of structures installed in a ship environment, and generate at least one corrected image by correcting distortion included in the at least one initial image using the three-dimensional position of the structure marker and the current two-dimensional position of the structure marker.

[0025] The advantages and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, but can be implemented in various different forms and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention. The embodiments presented below are provided to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. In describing the present invention, detailed descriptions of related known technologies are omitted if it is determined that such detailed descriptions may obscure the essence of the present invention.

[0026] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0027] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. Additionally, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as algorithms executed on one or more processors. Furthermore, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing, etc. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0028] Furthermore, the connecting lines or connecting members between the components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections. In the actual device, connections between components may be represented by various alternative or added functional connections, physical connections, or circuit connections.

[0029] The present disclosure will be described in detail below with reference to the attached drawings.

[0030] FIG. 1 is a conceptual diagram illustrating a method for generating a control solution using a control device in a ship environment according to one embodiment.

[0031] Here, the ship environment can refer to the interior or exterior of the ship (21), as well as any environment where the ship may be located, such as a shipbuilding workshop (22), a yard, and a port (23).

[0032] The control device (10) can detect events occurring in the ship environment. For example, the control device (10) can detect accidents, fires, drowning incidents, crowded areas, intrusion into dangerous areas, etc. occurring in the ship environment.

[0033] Additionally, the control device (10) can generate a control solution (30) to respond to events occurring in the ship environment. Additionally, the control device (10) can prevent the occurrence of events through various control solutions (30).

[0034] In one embodiment, the control device (10) can detect and track an object by creating a multi-channel environment area in a multi-channel environment where a plurality of shooting devices are installed. In addition, the control device (10) can predict the movement path of an object by analyzing the movement pattern of the object in the multi-channel environment.

[0035] In another embodiment, the control device (10) can improve the accuracy of object detection or event occurrence detection by correcting distortion in an image or video acquired from a camera installed in the ship environment. For example, the control device (10) can correct distortion using a specific structure or a virtual marker within the ship environment.

[0036] In another embodiment, the control device (10) can improve the accuracy of object detection by generating three-dimensional information of an object detected in an image or video acquired from a shooting device. For example, the control device (10) can generate three-dimensional information of an object using a neural network model or a projection matrix.

[0037] In another embodiment, the control device (10) may perform control by receiving images or video from only some of the multiple shooting devices installed in the ship environment to improve control efficiency. Specifically, the control device (10) may improve memory usage efficiency by deleting images or video from unselected shooting devices.

[0038] In another embodiment, the control device (10) may select object tracking information based on the amount of information change of an object detected in the ship environment and receive only the selected data. The control device (10) may use the received data to implement the ship environment where the object was detected using digital twin technology.

[0039] In another embodiment, the control device (10) can represent the ship environment using digital twin technology. Additionally, the control device (10) can create an interface that accurately displays the area where an event occurred or the area where an object was detected by implementing the ship environment in a three-dimensional form. Furthermore, the control device (10) can create and provide an interface to the user that displays the time when the event occurred or the time when the object was detected, as well as the type of the event or the type of the detected object. Through this, the user can quickly recognize the event or detected object that occurred within the ship environment and respond accordingly.

[0040] FIG. 2 is a block diagram of a control device according to one embodiment.

[0041] Referring to FIG. 2, the control device (100) includes a processor (110), memory (120), an input / output interface (130), and a communication module (140). For convenience of explanation, FIG. 2 only shows components related to the present invention. Accordingly, other general-purpose components may be included in the control device (100) in addition to the components shown in FIG. 2. Furthermore, it is obvious to those skilled in the art that the processor (110), memory (120), input / output interface (130), and communication module (140) shown in FIG. 2 may be implemented as independent devices.

[0042] The processor (110) can process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the instructions may be provided from memory (120) or an external device. Additionally, the processor (110) can generally control the operation of other components included in the control device (100).

[0043] The processor (110) can select a structure marker among a plurality of structures installed in the ship environment. For example, structures installed in the ship environment may refer to all equipment that can be installed in the ship environment, such as generators, engines, welding areas, containers, tanks, and propellers, but are not limited thereto.

[0044] Additionally, the processor (110) can obtain the three-dimensional position of the structure marker and the current two-dimensional position of the structure marker. For example, the processor (110) can obtain the initial two-dimensional position of the structure marker from an image or video in which the structure marker was captured, and then obtain the current two-dimensional position of the structure marker using characteristic information of the capturing device that captured the aforementioned image or video.

[0045] Additionally, the processor (110) can use the 3D position of the structure marker and the current 2D position to correct the distortion included in the initial image of the structure marker and generate at least a corrected image.

[0046] For example, the processor (110) can generate a projection matrix to correct image distortion using the 3D position of the structure marker and the current 2D position. Additionally, the processor (110) can generate a corrected image by correcting the initial image using the projection matrix.

[0047] The process performed in FIGS. 4 to 8 below may correspond to a process performed by a processor (110).

[0048] The processor (110) may be implemented as an array of multiple logic gates, or as a combination of a general-purpose microprocessor and memory storing a program that can be executed on the microprocessor. For example, the processor (110) may include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the processor (110) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor (110) may refer to a combination of processing devices such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a digital signal processor (DSP) core, or any other combination of such configurations.

[0049] The memory (120) may include any non-transient computer-readable recording medium. In one embodiment, the memory (120) may include a non-perishable permanent mass storage device such as RAM (random access memory), ROM (read only memory), disk drive, SSD (solid state drive), flash memory, etc. As another example, a non-perishable permanent mass storage device such as ROM, SSD, flash memory, disk drive, etc. may be a separate permanent storage device distinct from the memory. Additionally, the memory (120) may store an operating system (OS) and at least one program code (e.g., code for the processor (110) to perform an operation described later with reference to FIGS. 4 through 8).

[0050] These software components may be loaded from a computer-readable recording medium separate from the memory (120). This separate computer-readable recording medium may be a recording medium that can be directly connected to the control device (100), and may include, for example, an input / output computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. Alternatively, the software components may be loaded into the memory (120) through a communication module (140) that is not a computer-readable recording medium. For example, at least one program may be loaded into the memory (120) based on a computer program (e.g., a computer program for the processor (110) to perform the operation described later with reference to FIGS. 4 to 8) which is installed by files provided through the communication module (140) by developers or a file distribution system that distributes installation files for the application.

[0051] The input / output interface (130) may be a means for interfacing with a device for input and / or output (e.g., keyboard, mouse, etc.) that may be connected to or included in the control device (100). In FIG. 2, the input / output interface (130) is shown as an element configured separately from the processor (110), but is not limited thereto, and the input / output interface (130) may be configured to be included in the processor (110).

[0052] The communication module (140) may provide a configuration or function for the network control device (100) to communicate with an external device (not shown). Additionally, the communication module (140) may provide a configuration or function for the control device (100) to communicate with another external device. For example, control signals, commands, data, etc. provided under the control of the processor (110) may be transmitted to an external device via the communication module (140) and the network.

[0053] Additionally, although not shown in FIG. 2, the control device (100) may include a display module (not shown). For example, the display module may provide a screen to the user displaying three-dimensional information of an object detected in the second image and the type of the object.

[0054] FIG. 3 is an exemplary configuration diagram of a system including a control device according to one embodiment.

[0055] Referring to FIG. 3, the control device (310) may include any type of server that manages a web and / or app capable of providing artificial intelligence services. Additionally, the control device (310) of FIG. 3 may be the same device as the control device (10) of FIG. 1 and / or the device (100) of FIG. 2.

[0056] The external device (320) refers to an entity that provides information used by the control device (310) to correct distortion within the image. The external device (320) may include all types of servers that manage various types of information. The external device (320) may include a database and may include a server that manages a web service API capable of providing information, but is not limited thereto. For example, the external device (320) may correspond to a shooting device that transmits video or images of various areas within the ship environment to the control device.

[0057] The control device (310) and the external device (320) can communicate with each other and / or with other devices via a network. The network is a comprehensive data communication network that enables different entities to communicate smoothly with each other, and may include wired internet, wireless internet, and mobile wireless communication networks. For example, the network may include a Local Area Network (LAN), a Wide Area Network (WAN), a Value Added Network (VAN), a mobile radio communication network, a satellite communication network, and combinations thereof. Additionally, wireless communication may include, for example, Wi-Fi, Bluetooth, Bluetooth Low Energy, ZigBee, Wi-Fi Direct (WFD), Ultrawideband (UWB), Infrared Data Association (IrDA), Near Field Communication (NFC), but is not limited thereto.

[0058] The control device (310) can communicate with an external device (320) through a network. By communicating through the network, the control device (310) can receive data from the external device (320) and provide a response based on the received data.

[0059] FIG. 4 is a flowchart illustrating a method for correcting image distortion through the selection of a structure marker within a ship environment according to one embodiment.

[0060] Referring to FIG. 4, the method of correcting image distortion through the selection of a structural marker within the ship environment consists of steps processed sequentially in the control device (100) and / or processor (110) shown in FIG. 2. Therefore, even if details are omitted below, the description above regarding the control device (100) or processor (110) shown in FIG. 2 can also be applied to the method of correcting image distortion through the selection of a structural marker within the ship environment in FIG. 4.

[0061] In step 410, the control device can select a structure marker among a plurality of structures installed in the ship environment.

[0062] For example, structures installed in a ship environment may refer to all equipment that can be installed in a ship environment, such as generators, engines, welding areas, containers, tanks, and propellers, but are not limited thereto.

[0063] Here, a structure marker may refer to a marker set by a control device to correct distortion included in a video or image captured by a camera installed in the ship environment.

[0064] Considering the ship environment where many pieces of equipment are deployed, the control device may select structures with minimal positional change over time as structural markers to reliably correct distortion in video or images. Through this, the control device can reliably detect the occurrence of objects or events in the ship environment.

[0065] The method for selecting a structure marker will be explained in detail below with reference to Fig. 6.

[0066] In step 420, the control device can obtain the three-dimensional position of the structure marker and the current two-dimensional position of the structure marker.

[0067] For example, the control device can obtain the three-dimensional position of a structure marker using a structural drawing stored for the ship environment. Here, the structural drawing may refer to a CAD drawing.

[0068] In one embodiment, the control device can obtain information about the three-dimensional position of a structure selected as a structure marker in a CAD drawing of a ship. Here, the information about the three-dimensional position can be expressed in the form of (x, y, z).

[0069] In addition, the control device can acquire the current 2D position of the structure marker.

[0070] For example, the control device can acquire the initial two-dimensional position of a structure marker. In one embodiment, the control device can acquire the initial two-dimensional position of a structure marker using a structure drawing previously stored for the ship environment. Here, information regarding the two-dimensional position can be expressed in the form of (x, y).

[0071] In addition, the control device can obtain the current 2D position of the structure marker using the initial 2D position of the structure marker. In one embodiment, the control device can obtain the current 2D position of the structure marker using the initial 2D position of the structure marker and characteristic information of a camera that captured a video or image containing the structure marker.

[0072] The invention for acquiring current two-dimensional information will be described in detail below with reference to FIG. 8.

[0073] In step 430, the control device can generate at least one corrected image by correcting distortion included in at least one initial image of the structure marker captured using the 3D position and the current 2D position.

[0074] For example, the control device can generate a projection matrix to correct distortion included in at least one initial image of a structure marker using the 3D position and the current 2D position.

[0075] In addition, the control device can generate a corrected image using the 3D position of the structure marker, the current 2D position, and the projection matrix.

[0076] Referring to Fig. 5 below, a method for correcting image distortion through a projection matrix will be explained in detail.

[0077] FIG. 5 is an exemplary drawing for explaining a method for correcting distortion in an image of a ship environment according to one embodiment.

[0078] Referring to FIG. 5, the control device can obtain at least one initial image (510) of a structure marker (not shown) within the ship environment from a camera installed in the ship environment.

[0079] In the case of the initial image (510), distortion may be included due to various factors such as the installation location, magnification, and specifications of the shooting device. There was a problem that distortion included in the image could reduce the accuracy of object detection.

[0080] The control device can generate a corrected image (520) by correcting the distortion included in the initial image (510).

[0081] In one embodiment, the control device can correct distortions included in the initial image (510) using a projection matrix.

[0082] For example, the control device can calculate characteristic information of the imaging device using the 3D position of the structure marker and the current 2D position. Here, the characteristic information of the imaging device may include intrinsic parameters, extrinsic parameters, and distortion coefficients.

[0083] For example, a distortion coefficient may refer to a factor used to correct distortion caused by light entering unevenly across different regions through the lens of an imaging device. For instance, the distortion coefficient can be pre-set based on the specifications, zoom, angle of view, etc., of the imaging device.

[0084] Here, intrinsic parameters may refer to parameters related to distortion included in the image captured by the imaging device. Additionally, extrinsic parameters may refer to parameters related to the installation position and rotation of the imaging device.

[0085] For example, the control device can calculate a preliminary projection matrix using the following mathematical formula 1.

[0086]

[0087]

[0088] Here, (x, y) may represent the current 2D image coordinates of the structure marker. Additionally, (X, Y, Z) may represent the position of the structure marker in 3D space. Also, A can mean a preliminary projection matrix.

[0089] also, can mean an internal parameter. Also, can mean external parameters.

[0090] Specifically, f x can mean the focal length in the x-axis direction in pixels. Also, f y can refer to the focal length in the y-axis direction in pixels. Also, c x can mean the principal point in the x-axis direction. Also, c y can represent the principal point in the y-axis direction. Here, the principal point can represent the foot of the perpendicular line drawn from the center of the lens of the imaging device to the imaging device. Also, r represents the rotation state of the imaging device, that is, a rotation matrix representing the degree to which the axis of the imaging device has rotated relative to the 3D axis. Also, t represents the translation state of the imaging device, that is, a matrix representing the degree to which the axis of the imaging device has moved relative to the 3D axis. The control device can perform a first correction for distortion of the initial image (510) using the preliminary projection matrix. Additionally, the control device can re-acquire the current 2D position of the structure marker in the first corrected image generated through the first correction. Additionally, the control device can recalculate the intrinsic parameters using the aforementioned mathematical formula 1 using the 3D position of the structure marker and the re-acquired current 2D position. Additionally, the control device can calculate the final projection matrix using the recalculated intrinsic parameters and the preliminary projection matrix.

[0091] Additionally, the control device can use the final projection matrix to generate a corrected image (520) that corrects the distortion contained in at least one initial image (510) in which the structure marker is captured.

[0092] In another embodiment, the control device can correct distortions included in the initial image (510) using a neural network model.

[0093] In machine learning technology and cognitive science, a neural network model refers to a statistical learning algorithm implemented based on the structure of biological neural networks, or a structure that executes such an algorithm.

[0094] For example, a neural network model can represent a model capable of problem-solving by having nodes, which are artificial neurons forming a network through the combination of synapses as in biological neural networks, learn to reduce the error between the correct output corresponding to a specific input and the inferred output by repeatedly adjusting the weights of the synapses. For example, a neural network model may include random probability models and neural network models used in artificial intelligence learning methods such as machine learning and deep learning.

[0095] For example, a neural network model can be implemented as a multilayer perceptron (MLP) composed of multiple layers of nodes and connections between them. The neural network model according to the present embodiment can be implemented using one of various artificial neural network model structures including an MLP. For example, the neural network model may be composed of an input layer that receives an input signal or data from the outside, an output layer that outputs an output signal or data corresponding to the input data, and at least one hidden layer located between the input layer and the output layer, which receives a signal from the input layer, extracts a feature, and transmits it to the output layer. The output layer receives a signal or data from the hidden layer and outputs it to the outside.

[0096] Specifically, the control device can input an initial image (510) into a neural network model to obtain a corrected image (520) with the distortion corrected.

[0097] Additionally, the control device can train a neural network model. For example, the control device can train a neural network model using an initial image or initial video (510) of the ship environment as input data and a distortion-corrected corrected image or corrected video (520) as output data.

[0098] The control device can detect at least one object in a distortion-corrected image (520). Here, the object may refer to various equipment that may be located in the ship environment, such as people, vehicles, and structures, but is not limited thereto.

[0099] In addition, the control device can generate three-dimensional information about the detected object. Here, the three-dimensional information may include information about the location of the object within the ship environment and the space occupied by the object.

[0100] The control device can improve the accuracy of the three-dimensional information by generating three-dimensional information of an object using a distortion-corrected corrected image (520).

[0101] FIG. 6 is an exemplary drawing for explaining a method of selecting a structure marker according to one embodiment.

[0102] Referring to FIG. 6, the control device can acquire a plurality of initial images (610, 620) of the ship environment. Specifically, the control device can acquire a plurality of initial images (610, 620) from imaging devices installed in the ship environment.

[0103] In addition, the control device can acquire the time-series location of each of the multiple structures included in the ship environment. In addition, the control device can select a structure whose location does not change over time as a structure marker. Through this, the control device can effectively correct distortions included in the video (610, 620) or image (not shown) captured by the camera.

[0104] Specifically, the control device can determine that the reference structure (630) whose position has not changed corresponds to the first image (610) taken at the first time and the second image (620) taken at the second time by comparing the position of the structure (630) included in the first image (610) taken at the first time. Additionally, the control device can determine the reference structure (630) as a structure marker to correct distortion within the images (610, 620).

[0105] In one embodiment, the control device may select the center coordinates (not shown) of a reference structure (630) whose position does not change over time as a structure marker.

[0106] In another embodiment, the control device can detect the outline of a reference structure and select at least one point (631 to 636) included in the outline as a structure marker.

[0107] In another embodiment, the control device may select a structure marker using a neural network model. Specifically, the control device may obtain information about the structure marker by inputting at least one initial image (610, 620) of the ship environment into the neural network model. Additionally, the control device may use an image of the ship environment taken over a predetermined period as input data and use information about the structure marker included in the image as output data.

[0108] FIG. 7 is a flowchart illustrating a method for selecting a structure marker using time-based movement data of structures according to one embodiment.

[0109] Referring to FIG. 7, the method of selecting a structure marker using hourly movement data of structures consists of steps processed chronologically in the control device (100) and / or processor (110) shown in FIG. 2. Therefore, even if details are omitted below, the description above regarding the control device (100) or processor (110) shown in FIG. 2 can also be applied to the method of selecting a structure marker using hourly movement data of structures in FIG. 7.

[0110] In step 710, the control device can acquire multiple images of the ship environment.

[0111] For example, the control device can acquire multiple images from multiple imaging devices installed in the ship environment. Each imaging device can transmit images captured during a predetermined period to the control device. Additionally, each imaging device can transmit images captured at the same time to the control device.

[0112] In step 720, the control device can obtain the time-series location of each of the multiple structures included in the ship environment.

[0113] For example, the control device can detect at least one structure within a plurality of images. In one embodiment, the control device can detect a structure object within an image using an object detection model, which is a type of neural network model.

[0114] In addition, the control device can acquire the hourly location for each detected structure. For example, the control device can assign a unique ID to each detected structure and store information regarding the hourly location of each structure.

[0115] In step 730, the control device can calculate time-based movement data for each of the multiple structures.

[0116] For example, the control device can calculate time-based movement data for each of the multiple structures using the time-based positions of each of the multiple structures captured in multiple images.

[0117] Specifically, the control device can acquire the hourly position of each of the center positions of a plurality of structures and calculate the amount of position change per unit time as the hourly movement amount data described above.

[0118] In step 740, the control device can select a structure marker based on each hourly movement data.

[0119] In one embodiment, the control device can select the structure with the smallest hourly movement data among a plurality of structures as a structure marker.

[0120] In another embodiment, the control device selects only the structure among a plurality of structures for which the hourly movement amount data is 0 as a structure marker, and if there is no structure for which the position change amount per unit time is 0, it can re-acquire another image of the ship environment.

[0121] The control device can set the size of a structure as a constraint for selecting a structure marker. For example, the control device can set a constraint that the size of the structure selected as a structure marker must exceed a preset threshold. Through this, the control device can prevent minute movements, etc., caused by the small size of the structure selected as a structure marker.

[0122] FIG. 8 is an exemplary drawing for explaining a method of generating a correction image using the current two-dimensional position of a structure marker according to one embodiment.

[0123] Referring to FIG. 8, the method for generating a correction image using the current two-dimensional position of a structure marker consists of steps processed chronologically in the control device (100) and / or processor (110) shown in FIG. 2. Therefore, even if details are omitted below, the description above regarding the control device (100) or processor (110) shown in FIG. 2 can also be applied to the method for generating a correction image using the current two-dimensional position of a structure marker in FIG. 8.

[0124] In step 810, the control device can acquire the initial two-dimensional position of the structure marker.

[0125] Here, information about the 2D position can be expressed in the form of (x, y).

[0126] In one embodiment, the control device can obtain the three-dimensional position of a structure marker using a structural drawing stored for the ship environment. Here, the structural drawing may refer to a CAD drawing.

[0127] For example, the control device can obtain the 3D position of a structure marker set through the absolute origin coordinates of the ship environment using a CAD drawing. Here, the absolute origin coordinates may refer to coordinates having the pre-set 3D coordinates of (0,0,0).

[0128] In another embodiment, the control device can acquire the initial two-dimensional positions of structures within the ship environment stored in each imaging device. Each imaging device can store initial characteristic information and the initial two-dimensional positions of structures located in the area within the ship environment that the imaging device is capturing.

[0129] In step 820, the control device can obtain the current 2D position using characteristic information of the camera that captured at least one initial image.

[0130] For example, characteristic information of the imaging device may include information regarding the tilt, rotation, and magnification of the imaging device, but is not limited thereto.

[0131] The control device can obtain the current 2D position from the initial 2D position of the structure marker by comparing the initial characteristic information and current characteristic information of the imaging device.

[0132] In one embodiment, the control device can obtain a current two-dimensional position from an initial two-dimensional position based on at least one of information regarding the current tilt relative to the initial tilt of the imaging device, the current rotation relative to the initial rotation, and the current magnification relative to the initial magnification.

[0133] In another embodiment, the control device can acquire the current 2D position of a structure marker using an area tracking algorithm such as an Optical Flow Algorithm. Specifically, the control device can acquire the current 2D information of the structure marker by extracting changes in pixel positions from an image that changes over time using an Optical Flow Algorithm.

[0134] In step 830, the control device can generate at least one corrected image by correcting distortion included in at least one initial image of a structure marker using the 3D position and the current 2D position.

[0135] For example, the control device can generate a corrected image by correcting the distortion included in the initial image through the method of correcting distortion within the image in FIG. 5 described above.

[0136] An embodiment according to the present invention may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. In this case, the medium may include a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a hardware device specifically configured to store and execute program instructions, such as a ROM, RAM, or flash memory.

[0137] Meanwhile, the above-mentioned computer program may be one specifically designed and configured for the present invention or one known and available to those skilled in the art of computer software. Examples of computer programs may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0138] According to one embodiment, the method according to various embodiments of the present disclosure may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two users and devices. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0139] Unless explicitly stated otherwise regarding the order of the steps constituting the method according to the present invention, said steps may be performed in a suitable order. The present invention is not necessarily limited by the order in which said steps are described. The use of all examples or exemplary terms (e.g., etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is not limited by said examples or exemplary terms unless limited by the claims. Furthermore, those skilled in the art will understand that various modifications, combinations, and changes may be made according to design conditions and factors within the scope of the claims or equivalents to which they are added.

[0140] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention.

Claims

1. A step of selecting a structure marker among a plurality of structures installed in the ship environment; A step of obtaining the 3D position of the structure marker and the current 2D position of the structure marker; and A step of generating at least one corrected image by correcting distortion included in at least one initial image captured of the structure marker using the above 3D position and the above current 2D position; A method for correcting image distortion by selecting a structural marker within a ship environment, including 2. In Paragraph 1, The step of selecting the above-mentioned structure marker is, A step of acquiring a plurality of initial images of the above-mentioned ship environment; A step of obtaining the time-sequence location of each of the plurality of structures included in the above-mentioned ship environment; and A step of selecting the structure marker using each of the above time-series locations; A method including 3. In Paragraph 2, The step of selecting the above-mentioned structure marker is, A step of calculating time-based movement data for each of the plurality of structures; and A step of selecting the structure marker based on the above-mentioned time-based movement data; A method including 4. In Paragraph 1, The step of selecting the above-mentioned structure marker is, A step of selecting a reference structure among the plurality of structures above; A step of detecting the outline of the above reference structure; and A step of selecting at least one point included in the above outline as the structure marker; A method including 5. In Paragraph 1, The step of selecting the above-mentioned structure marker is, A step of selecting a reference structure among the plurality of structures above; A step of detecting the center point of the above reference structure; and A step of selecting the above center point as the above structure marker; A method including 6. In Paragraph 1, The above-mentioned acquisition step is, A method for obtaining the three-dimensional position using a previously stored structural drawing for the above-mentioned ship environment.

7. In Paragraph 1, The above-mentioned acquisition step is, A step of obtaining an initial two-dimensional position of the above-mentioned structure marker; and A step of obtaining the current 2D position using characteristic information of a shooting device that captured at least one initial image; Includes, The above characteristic information is, A method comprising at least one of placement information of a shooting device, shooting information, and specification information.

8. In Paragraph 1, The step of generating at least one correction image is, A step of calculating a projection matrix using the above 3D position and the above current 2D position; and A step of generating at least one corrected image using the above projection matrix; A method including 9. In Paragraph 1, The above method is, A step of detecting at least one object in the above at least one corrected image; and A step of generating three-dimensional information for at least one object; A method including 10. In Paragraph 1, The step of selecting the above-mentioned structure marker is, Step of setting constraints on the size of the structure; and A step of selecting a structure satisfying the constraint among the plurality of structures above as the structure marker; A method including 11. In Paragraph 1, The above-mentioned acquisition step is, A method for obtaining the three-dimensional position based on absolute origin coordinates set within the above-mentioned ship environment.

12. In Paragraph 2, The step of selecting the above-mentioned structure marker is, A method for selecting a structure that remains unchanged at the above-mentioned time-based position as the above-mentioned structure marker.

13. In Paragraph 1, The above plurality of structures are, It includes at least one of the generator and power device within the above-mentioned vessel, and The step of selecting the above-mentioned structure marker is, A method of selecting at least one vertex included in each of the plurality of structures as the structure marker.

14. At least one memory; and Includes at least one processor; and The above-mentioned at least one processor is, A computing device that selects a structure marker using movement data for each of a plurality of structures installed in a ship environment, and generates at least one corrected image by correcting distortion included in at least one initial image using the 3D position of the structure marker and the current 2D position of the structure marker.

15. A computer-readable recording medium storing a program for executing the method according to paragraph 1 on a computer.

Images