Determining the location of objects on the basis of image data

Abstract

The invention relates to methods for localizing objects on the basis of digital image data. The present method for determining the location of objects in a surrounding environment on the basis of image data comprises receiving an image, captured by a recording device, which contains an object in the surrounding environment. The received image is processed and objects in the image are identified. The identified objects in the image are then localized on the basis of data obtained through the following steps: determining the geo-position of the point of capture of the image; determining the angle at which each identified object is situated in relation to the point of capture, on the basis of the data of the azimuth of the capture and the horizontal location of each identified object in the image; determining the distance to each identified object in the image from the point of capture; displaying the localized objects on a digital map. The invention makes it possible to improve the accuracy in determining the location of objects in digital images taken of a surrounding environment.

Description

PCI7RU2025 / 000008 METHOD AND SYSTEM FOR DETERMINING THE LOCATION OF OBJECTS IN THE SURROUNDING SPACE BASED ON IMAGE DATA. TECHNICAL FIELD

[0001] This technical solution relates to the field of information technology, in particular to methods of localizing objects using digital image data. LEVEL OF TECHNOLOGY

[0002] Mapillary, a community-driven application for creating a database of geotagged images, is known from the prior art (see https: / / en.wikipedia.org / wiki / Mapillary). This solution is a service that allows for the creation of digital maps based on real-world images of the surrounding area. Each image is geotagged by the user, allowing for the relative accuracy of determining the location of the imaged area within the coordinate grid of the digital map.

[0003] This principle has a relatively low accuracy of localization of the objects themselves, such as points of interest (POI), buildings, parks, etc., since it does not involve the identification of such objects and their localization for more accurate construction of digital maps, which can subsequently be used for the purposes of determining user locations, navigation, etc. ESSENCE OF THE INVENTION

[0004] The claimed invention allows solving the technical problem of localizing objects in digital images of the surrounding space.

[0005] The technical result is an increase in the accuracy of determining the location of objects in digital images of the surrounding space.

[0006] In a preferred embodiment, a method is claimed for determining the location of objects in the surrounding space using image data, comprising the steps of: a) at least one image is obtained, recorded by the shooting device, containing at least one object of the surrounding space; b) process the obtained image using an ensemble of machine learning models, each of which is designed to recognize a specific type of object in the surrounding space; c) determine all objects in the surrounding space in the image; d) determine the geo-position of the image shooting point; e) determine the angle at which each detected object is located relative to the survey point, based on the survey azimuth data and the horizontal location of each detected object in the image; f) determine the distance to each identified object in the image from the shooting point; g) localize certain objects in the image based on the data obtained in steps d) - f); h) display localized objects on a digital map.

[0007] In one of the particular examples of the method implementation, the geo-coordinates of localized objects are determined.

[0008] In another particular example of the implementation of the method, the location of the shooting device is additionally determined.

[0009] In a preferred embodiment, a system for determining the location of objects in the surrounding space based on image data is claimed, comprising at least one processor connected to at least one memory storing machine-readable instructions which, when executed by the processors, implement the above-mentioned method. BRIEF DESCRIPTION OF DRAWINGS

[0010] Fig. 1 illustrates the general scheme of the claimed solution. [UN] Fig. 2 illustrates the sequence of steps in executing the method for determining location.

[0012] Fig. 3 illustrates the general appearance of the computing device. IMPLEMENTATION OF THE INVENTION

[0013] As shown in Fig. 1, the claimed solution is a client-server architecture, in which the client is a user mobile device (MD) that exchanges data with a server (120). The user device may be a smartphone, tablet, or any other device that is equipped with a camera and is capable of photographing the surrounding space (130) with the receipt of images (131) transmitted via a data transmission channel to the server (120), on which localization of objects (1311 - 1313) in the image is subsequently carried out.

[0014] The data transmission channel can be an Internet connection provided via a cellular network (GSM, 3G, 4G, 5G, etc.), or by connecting to access points via wireless communication protocols (WiFi, WLAN, etc.).

[0015] Fig. 2 shows a flow chart of the claimed method (200) for determining the locations of objects in digital photographs. In the first step (201), an image (131) of the surrounding space (130), such as a building, a point of interest, etc., is obtained using the user's device (110). The radio signal strength at the point of survey may be additionally determined for the obtained image (131). Publicly available signals may be used to obtain radio signal data, for which a unique identifier is stored, such as communication tower data, the type of radio signal, and its strength at a specific point at a specific time. This function may be implemented using a specialized application, which may be used to perform surveys and simultaneously determine radio signal data at the point of survey.

[0016] The received data is transmitted at step (202) to the server (120) for subsequent processing. The image received from the user device (110) is vectorized and transmitted to an ensemble of machine learning models, each of which is configured to recognize a specific type of object in the surrounding space. Specifically, a model for recognizing buildings, structures (monuments, monuments, etc.), trees, fences, signs, etc. can be used. The specific set of models can be modified and supplemented depending on the type of objects to be recognized in the images.

[0017] For each image, the geolocation (geocoordinates) of the shooting point is also determined at step (203). This function can be implemented by obtaining data from the application from which the shooting is performed, as well as from the GNSS data of the user's device (110).

[0018] At step (204), the angle at which each detected object (1311 - 1313) is located relative to the shooting point is determined based on the shooting azimuth data (the direction of the camera when taking the picture) and the horizontal position of each detected object (1311 - 1313) in the image (131).

[0019] After obtaining the angle value in step (204), the distance at which each of the determined objects (1311 - 1313) is located from the shooting point is calculated in step (205). This can be implemented by using known values the dimensions of the object type (for example, the size of road signs, buildings, the average size of a type of tree, etc.), for example, using the angular size.

[0020] Another example might not rely on the size of objects in the image. In this case, a neural network is used to determine the depth of an image in a selected area (e.g., DPT - Dense Prediction Transformer).

[0021] At step (206), the final localization of objects (1311 - 1313) on the digital map is performed based on the processing of the input image (131). Based on the obtained information about the survey point and the coordinates of this point and the information obtained in steps (204) and (205), a geocoordinate is assigned to each object (1311 - 1313). This can be accomplished by determining the coordinates using polar coordinates.

[0022] As a result, geocoordinates are indicated on the digital map for certain types of objects. The resulting geocoordinate information can be transmitted to the user's device's mobile app interface (110) and used, for example, to determine the user's location, for navigation purposes, etc.

[0023] Additionally, certain objects can be deduplicated from images received from users, allowing for the unambiguous identification of a specific object based on multiple images. Since the same object often appears in multiple images (photographs), each such object is represented by a set of points of the same type in close proximity. Using a clustering method for objects of the same type, clusters of points characterizing a specific object are identified. After clustering, the most probable location of the object is determined based on the obtained coordinates.

[0024] Fig. 3 shows a general view of a computing system implemented on the basis of a computing device (300). In the general case, the computing device (300) comprises one or more processors (301), memory means such as RAM (302) and ROM (303), input / output interfaces (304), input / output devices (305), and a device for network interaction (306), united by a common information exchange bus.

[0025] The processor (301) (or several processors, a multi-core processor) may be selected from a range of devices that are widely used today, such as those from Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™, Qualcomm Snapdragon™, etc. The processor also includes a graphics processor, such as an NVIDIA or ATI GPU, which is also suitable for fully or partially implementing the method (200). The memory means may be the available memory capacity of a graphics card or graphics processor.

[0026] RAM (302) is random access memory (RAM) and is designed to store machine-readable instructions executed by the processor (301) to perform the necessary logical data processing operations. RAM (302) typically contains executable instructions from the operating system and corresponding software components (applications, software modules, etc.).

[0027] ROM (303) is one or more permanent data storage devices, such as a hard disk drive (HDD), a solid-state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

[0028] To organize the operation of the device components (300) and to organize the operation of external connected devices, various types of I / O interfaces (304) are used. The choice of the appropriate interfaces depends on the specific design of the computing device, which may include, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0029] To ensure user interaction with the computing device (300), various I / O information means (305) are used, for example, a keyboard, display (monitor), touch display, touchpad, joystick, mouse, light pen, stylus, touch panel, trackball, speakers, microphone, augmented reality means, optical sensors, tablet, light indicators, projector, camera, biometric identification means (retina scanner, fingerprint scanner, voice recognition module), etc.

[0030] The network interaction means (306) ensures the transmission of data by the device (300) via an internal or external computer network, for example, the Intranet, the Internet, a LAN, etc. One or more means (306) may be, but are not limited to: an Ethernet card, a GSM modem, a GPRS modem, an LTE modem, a 5G modem, a satellite communication module, an NFC module, a Bluetooth and / or BLE module, a Wi-Fi module, etc.

[0031] Additionally, satellite navigation tools included in the device (300) can also be used, for example, GPS, GLONASS, BeiDou, Galileo.

[0032] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims

FORMULA 1. A method for determining the location of objects in the surrounding space using image data, comprising the following steps: a) at least one image is obtained, recorded by the shooting device, containing at least one object of the surrounding space; b) process the obtained image using an ensemble of machine learning models, each of which is designed to recognize a specific type of object in the surrounding space c) identify all objects in the surrounding space in the image; d) determine the geo-position of the image shooting point; e) determine the angle at which each detected object is located relative to the survey point, based on the survey azimuth data and the horizontal location of each detected object in the image; f) determine the distance to each identified object in the image from the shooting point; g) localize certain objects in the image based on the data obtained in steps d) - f); h) display localized objects on a digital map.

2. The method according to paragraph 1, characterized in that the geo-coordinates of localized objects are determined.

3. The method according to paragraph 2, characterized in that the location of the filming device is additionally determined.

4. A system for determining the location of objects in the surrounding space based on image data, comprising at least one processor connected to at least one memory storing machine-readable instructions that, when executed by the processors, implement the method according to any one of paragraphs 1-3.

Images