Method and system for assisting the movement of a person with visual impairment

Abstract

The invention relates to a method for assisting the movement of a person with visual impairment, the method comprising a step (101) of acquiring spatial data (D), and a step (102) of generating a pixelated image (I PIX ). The method comprises, for each pixel, a step (103) of determining coordinates of the pixel, a step (104) of determining a depth (Z) of the pixel and a step (105) of determining a danger level (Li) of the pixel. The method further comprises a step (107) of determining a direction (Dir) of movement and a step (108) of audio-guiding the person with visual impairment according to the determined direction (Dir) of movement.

Description

[0001] DESCRIPTION

[0002] Title of the invention: Method and system for assisting the movement of a visually impaired person

[0003] [1] Technical field

[0004] [2] The present invention relates to the field of mobility assistance for visually impaired persons. In particular, the invention relates to a method and a portable system for mobility assistance for such persons. The term "visually impaired person" here encompasses both moderately visually impaired persons and persons with severe visual impairments (such as blindness).

[0005] [3] Prior Art

[0006] [4] According to a recent study, more than 335 million people worldwide have severe visual impairment, of whom 42 million are blind. These individuals require assistance from a third party to get around or the help of an animal. Guide dogs play a crucial role in this area. Unfortunately, such dogs are a scarce resource, and few people can benefit from them. Furthermore, dog allergies also hinder their use.

[0007] [5] Numerous technological solutions exist to address the mobility challenges faced by visually impaired individuals. One such solution is based on ultrasonic obstacle detection integrated into a white cane. Sweeping the cane detects obstacles and returns the reflected ultrasonic signal. A sound is then emitted by a speaker built into the cane or by a Bluetooth™-type earpiece to inform the user of the obstacle's presence.

[0008] [6] A solution based on a helmet positioned on the head of the person being helped, which allows for assistance without a cane, is also known. However, the use of such a helmet is not very intuitive for the user, and its weight is also a deterrent to its use.

[0009] [7] There is therefore a need to provide a mobility assistance solution for visually impaired people that is simple to implement and practical to use. [8] Summary of the invention

[0010] [9] A first object of the invention relates to a method for assisting visually impaired persons, said method comprising:

[0011] - a step of acquiring, by a camera, spatial data of an environment in which the visually impaired person is able to move around;

[0012] - a step of generating a pixelated image from said spatial data, said pixelated image comprising a plurality of pixels;

[0013] - for each pixel in the plurality of pixels:

[0014] - a step to determine the coordinates of the pixel;

[0015] - a step to determine the depth of the pixel;

[0016] - a step to determine a level of danger of the pixel;

[0017] - a step of generating a 3D point cloud, each point of the point cloud being associated with a pixel of the pixelated image, each point of the point cloud being positioned in the point cloud by coordinates of the point, said coordinates of the point being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being affected by the level of danger of said pixel;

[0018] - a step of determining a direction of movement for the visually impaired person from the coordinates of all or part of the points in the 3D point cloud and the danger levels assigned to said points;

[0019] - an audio guidance step for the visually impaired person according to the determined direction of movement.

[0020]

[0010] The invention thus makes it possible to model and describe the user's environment. This environment is constituted as a three-dimensional geometric point cloud. The invention proposes using a single, very low-cost camera and constructing the 3D environment as a point cloud, each point of which has three metric coordinates along the X, Y, and Z axes relative to the camera, from a single image, by assigning, on the one hand, an object class semantics, and on the other hand, a danger semantics to each point of the 3D representation.

[0021]

[0011] The invention thus makes it possible to guide a person with a visual impairment using spatial sound in order to avoid obstacles by indicating a safe direction of movement to follow.

[0022]

[0012] In an alternative embodiment, the coordinates of the point are determined

[0023] X' -u- according to the following equation: Y' = [K -1 ] * A v with A a scale factor related to the Z-1- .1. camera type and [ / <] an intrinsic camera matrix, such that: [ <] = with fx a camera focal length in number of pixels along a first direction x, fy a focal length of the camera in number of pixels along a second direction y, Cx a first coordinate in pixels of the intersection of an optical axis of the camera with an image plane, Cy a second coordinate in pixels of the intersection of the optical axis of the camera with said image plane.

[0024]

[0013] In one embodiment, the step of determining the danger level of the pixel includes a step of determining for said pixel a pixel semantics, said pixel danger level being obtained from said pixel semantics.

[0025]

[0014] In an embodiment, each point of the 3D point cloud having a height relative to a ground, the step of determining a direction of movement includes a step of eliminating points from the 3D point cloud whose height is greater than a certain value, to form a corrected 3D point cloud.

[0026]

[0015] In one embodiment, said elimination step takes into account an inclination and / or an irregularity of the ground.

[0016] In one embodiment, the step of determining a direction of movement includes a step of projecting the points of the corrected 3D point cloud onto the ground in order to determine the direction of movement.

[0027]

[0017] In one embodiment, the audio guidance step for the visually impaired person according to the determined direction of movement includes a step for generating a spatial 3D sound.

[0028]

[0018] In one embodiment, the audio guidance step includes a step for correcting the generated spatial 3D sound, said correction being carried out from a 3x3 rotation matrix R and a 1x3 translation matrix T of the camera so as to guide the person independently of the rotation and translation movements of said camera.

[0029]

[0019] In one embodiment, the determination of the level of danger of the pixel is carried out from a general classification of the environment in which the visually impaired person is able to move, said classification being obtained from a third artificial intelligence module and the semantics of each pixel determined by a second artificial intelligence module.

[0030]

[0020] In one embodiment, the depth Z in metric distance per pixel is determined by a first artificial intelligence module, the semantics of each pixel is determined by a second artificial intelligence module and the general classification of the environment by a third intelligence module are unified in a single artificial intelligence module.

[0031]

[0021] A unified module, that is, a module capable of determining the metric depth, segmentation, and classification of the environment, makes it possible to provide a single image and directly collect all the required information, namely the 4D representation (3 distance dimensions and the hazard level). This allows for increased efficiency in input / output operations compared to a solution with several separate modules. Indeed, by sharing the image feature extraction process, it is possible to simultaneously save time, memory, and improve accuracy.

[0032]

[0022] Another object of the invention relates to a system for assisting the movement of a visually impaired person, said system comprising a portable device intended to be positioned on the head of said person and a processing unit, said portable device comprising a camera adapted to acquire spatial data of an environment in which the visually impaired person is able to move, said portable device comprising an audio module for providing audio guidance to the visually impaired person along a direction of movement determined from the spatial data, said system comprising at least one processor configured to implement:

[0033] - a step of acquiring spatial data using the camera;

[0034] - a step of generating a pixelated image from the spatial data, said pixelated image comprising a plurality of pixels; for each pixel of the plurality of pixels:

[0035] - a step of determining the coordinates of the pixel in the pixelated image, called pixel coordinates;

[0036] - a step to determine the depth of the pixel;

[0037] - a step to determine a level of danger of the pixel;

[0038] - a step of generating a 3D point cloud, each point of the point cloud being associated with a pixel of the pixelated image, each point of the point cloud being positioned in the point cloud by coordinates of the point, said coordinates of the point being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being affected by the level of danger of said pixel;

[0039] - a step of determining a direction of movement for the visually impaired person from the coordinates of all or part of the points in the 3D point cloud and the danger levels assigned to said points;

[0040] - an audio guidance step, using the audio module, for the visually impaired person in the determined direction of movement.

[0023] Another object of the invention relates to a computer program comprising software instructions which, when executed by a computer, implement a method for assisting the movement of a visually impaired person.

[0041]

[0024] Another object of the invention relates to a method for acquiring spatial data to assist the movement of a visually impaired person, said method comprising:

[0042] - a stage of acquiring spatial data using a camera in which the visually impaired person is able to move around;

[0043] - a step of generating a pixelated image from said spatial data, said pixelated image comprising a plurality of pixels; said spatial data being intended to be able to implement, for each pixel of the plurality of pixels;

[0044] - a step of determining the coordinates of the pixel in the acquired image;

[0045] - a step to determine the depth of the pixel;

[0046] - a step to determine a level of danger for said person;

[0047] - a step of generating a 3D point cloud, each point of the point cloud being associated with a pixel of the pixelated image, each point of the point cloud being positioned in the point cloud by coordinates of the point, said coordinates of the point being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being affected by the level of danger of said pixel;

[0048] - a step of determining a direction of movement for the visually impaired person from the coordinates of all or part of the points in the 3D point cloud and the danger levels assigned to said points;

[0049] - said acquisition method further comprising an audio guidance step for the visually impaired person in the determined direction of movement.'

[0025] Another object of the invention relates to a portable device intended to be positioned on the head of a visually impaired person, said portable device comprising:

[0050] - a camera adapted to acquire spatial data of an environment in which the visually impaired person is able to move around and to generate a pixelated image from said spatial data, said pixelated image comprising a plurality of pixels;

[0051] - an audio module for audio guidance of the visually impaired person along a determined direction of movement based on spatial data;

[0052] - a processor configured to implement a step of acquiring spatial data by the camera, said spatial data being intended to be implemented by said processor and / or by another processor, for each pixel of the plurality of pixels:

[0053] - a step to determine the coordinates of the pixel;

[0054] - a step to determine the depth of the pixel;

[0055] - a step to determine a level of danger of the pixel;

[0056] - a step of generating a 3D point cloud, each point of the point cloud being associated with a pixel of the pixelated image, each point of the point cloud being positioned in the point cloud by coordinates of the point, said coordinates of the point being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being affected by the level of danger of said pixel;

[0057] - a step of determining a direction of movement for the visually impaired person from the coordinates of all or part of the points in the 3D point cloud and the danger levels assigned to said points;

[0058] - said processor and / or said other processor being configured to implement audio guidance (108) for the visually impaired person in the determined direction (Dir) of movement.

[0026] Another object of the invention relates to a computer program comprising software instructions which, when executed by a computer, implement a method for acquiring an image to assist the movement of a visually impaired person.

[0059]

[0027] Another object of the invention relates to a treatment method for assisting the movement of a visually impaired person, said method comprising:

[0060] - a step of generating a 3D point cloud, each point of the point cloud being associated with a pixel of a pixelated image obtained from spatial data via a camera, each point of the point cloud being positioned in the point cloud by coordinates of the point, said coordinates of the point being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being assigned a danger level of said pixel, the coordinates of the pixel, the depth of the pixel, the danger level of said pixel being obtained from said spatial data;

[0061] - a step of determining a direction of movement for the visually impaired person from the coordinates of all or part of the points in the 3D point cloud and the danger levels assigned to said points, said determined direction allowing audio guidance of the person.

[0062]

[0028] Another object of the invention relates to a data processing device for assisting the movement of a visually impaired person, said processing device comprising a processor configured to implement:

[0063] - a step of generating a 3D point cloud, each point of the point cloud being associated with a pixel of a pixelated image obtained from spatial data via a camera, each point of the point cloud being positioned in the point cloud by point coordinates, said point coordinates being determined from the coordinates of the pixel associated with said point and from the depth of said pixel, said point of the point cloud being assigned a danger level of said pixel, the pixel coordinates, the pixel depth, the danger level of said pixel being obtained from said spatial data; - a step of determining a direction of movement for the visually impaired person from the coordinates of all or part of the points of the 3D point cloud and the danger levels assigned to said points, said determined direction allowing audio guidance of the person.

[0064]

[0029] Another object of the invention relates to a computer program comprising software instructions which, when executed by computer, implement a data processing method for assisting the movement of a person with a visual impairment.

[0065]

[0030] Another object of the invention relates to a portable device intended to be positioned on the head of a visually impaired person, said portable device comprising:

[0066] - a main body;

[0067] - a first branch;

[0068] - a second branch, the said first branch and the said second branch extending the said main body;

[0069] - said portable device comprising:

[0070] - a camera on the first branch, said camera being a pinhole camera;

[0071] - a first speaker on the first branch and a second speaker on the second branch, said first speaker and said second speaker being capable of generating a 3D spatial sound for the audio guidance of the visually impaired person along a predetermined direction of movement;

[0072] - a human-machine interface dial for controlling the portable device by the visually impaired person or by voice.

[0073]

[0031] In one embodiment, the first branch comprises a head, a body, and an intermediate portion between said head and said body, and the second branch comprises a head, a body, and an intermediate portion between said head and said body.

[0032] In one embodiment, the head of the first branch comprises the camera and the human-machine interface wheel.

[0074]

[0033] In one embodiment, the first loudspeaker and the second loudspeaker are respectively arranged on the first branch and on the second branch, opposite each other.

[0075]

[0034] In one embodiment, the device is capable of implementing part of a method for assisting the movement of a person with a visual impairment.

[0076]

[0035] In one embodiment, the device is part of a system for assisting the mobility of a visually impaired person

[0077]

[0036] Description of the figures

[0078]

[0037] Other features and advantages of the invention will become apparent upon reading the detailed description that follows, for an understanding of which reference should be made to the accompanying drawings in which:

[0079]

[0038] Figure 1 illustrates in general terms a method for assisting the movement of a person with a visual impairment according to the invention;

[0080]

[0039] Figure 2 illustrates a geometric visualization of spatial data of an environment in which the visually impaired person is able to move around;

[0081]

[0040] Figure 3 illustrates a pixelated image I PEX obtained from the spatial data in Figure 2;

[0082]

[0041] Figure 4 illustrates in more detail a step in determining a hazard level of the process of Figure 1;

[0083]

[0042] Figure 5 illustrates in more detail a step of determining a direction of movement of the process of Figure 1;

[0043] Figure 6 illustrates in more detail a step of audio guidance of the visually impaired person of the process of Figure 1;

[0084]

[0044] Figure 7 illustrates a system for assisting the movement of a person with a visual impairment for the implementation of the method of Figure 1;

[0085]

[0045] Figure 8 illustrates a portable device of the system of Figure 7 according to a first view;

[0086]

[0046] Figure 9 illustrates a portable device of the system of Figure 7 according to a second view;

[0087]

[0047] Figure 10 illustrates an internal operation of the portable device of Figures 8 and 9;

[0088]

[0048] Figure 11 illustrates an internal operation of a processing device of the system of Figure 7.

[0089]

[0049] Description of embodiments

[0090]

[0050] Figure 1 illustrates the overall operation of a method for assisting the movement of a person with a visual impairment.

[0091]

[0051] This method includes an acquisition step 101, by a camera, of spatial data D of an environment in which the visually impaired person is able to move around and a generation step 102 of a pixelated image from the spatial data D.

[0092]

[0052] The camera here is a pinhole camera, for example a standard monocular camera. It is a simple device that offers many advantages in terms of manufacturing cost.

[0093]

[0053] The pinhole camera is thus capable of capturing a plurality of points M 3D constituting the spatial data D. By the process of the invention, it is possible to obtain a point M^ iV belonging to the pixelated image I PEX from one of the points M 3Dspatial data D via a linear transformation (called "mapping" in English).

[0054] Such a linear transformation is illustrated in particular from Figures 2 and 3.

[0094]

[0055] Figure 2 shows an orthogonal coordinate system with axes X, Y, Z centered at O, O being the optical center of the camera.

[0095]

[0056] In such a Euclidean space, each point M XIY can be defined by a triplet of coordinates such that: M 3D where X is the coordinate of M 3D along the X, Y axis is the coordinate of M 3D along the F axis and Z is the coordinate of M 3D along the Z-axis.

[0096]

[0057] By fixing a value for Z, for example the value 1, it is possible to determine a plane P passing through the point M X Y - This plane P corresponds to the image that the camera can acquire.

[0097]

[0058] In this plane P visible in figure 2, the point M 3Dpresents the coordinates

[0098] X following: M 3D Y

[0099] .1.

[0100]

[0059] It is also possible to determine homogeneous coordinates of this point M 3D by adding an additional parameter to the coordinate triplet, for example a constant, such as 1. The homogeneous coordinates of such a point

[0101] X'

[0102] Y can then be written as: MSD = Z .1.

[0103]

[0060] As illustrated in Figure 3, the image generated in step 102 is a pixelated image I PEX obtained from spatial data D, in particular those present on plane P.

[0104]

[0061] The pixelated image I PEX is divided into a plurality of Pixi pixels, i ranging from 1 to N. Each Pixi pixel here has the shape of a square with a center M u ' v .

[0105]

[0062] In a step 103, the coordinates of the pixel Pixi in the pixelated image I are determined PEX , say pixel coordinates (u, v, 1);

[0106]

[0063] The coordinates of point M u ' v are thus determined in a coordinate system with axes Ü, V and center O'. The center O' is represented here at the center of the pixelated image fpEX-

[0064] The linear transformation between the point M 3D and point M u ' v This is done using the following equations:

[0107]

[0066] Where:

[0108]

[0067] fx denotes a focal length of the camera in number of pixels along the direction

[0109]

[0068] fy denotes the focal length of the camera in number of pixels along the direction Ÿ;

[0110]

[0069] Cx and Cy denote coordinates of an intersection C of an optical axis

[0111] AO with plane P passing through point M 3DThe optical axis AO extends the Z axis in Figure 2. The intersection C between the optical axis AO and the plane P is positioned at the center of this plane.

[0112]

[0070] l is a scaling factor that links the relative consistency of a distribution in the Z-axis for a given image.

[0113]

[0071] The matrix [ <] is called the intrinsic matrix of the pinhole camera. This matrix is ​​calculated from the characteristics of the camera.

[0114]

[0072] Thus, this matrix [K] allows the linear transformation (mapping) of a point in the 3D space of the real world defined by (X, Y, Z, 1) T on the image plane from an image sensor of the camera (u, v, 1) T .

[0115]

[0073] From equation 1, the following 3 relationships can be defined (Equations 2):

[0116]

[0074] Zu = X * f x + Z * C x ;

[0117]

[0076] Z = Z.

[0118]

[0077] After simplification we have the following system of equations (Equations 3):

[0119]

[0078] u = X * + C x ;

[0079] v = Y*j f + C y .

[0120]

[0080] Equations 3 thus allow the passage from a 3D point (real world) to 2D (image).

[0121]

[0081] The process in Figure 1 also includes a determination step

[0122] 104 of a depth Z for each Pixi pixel. By "depth Z," we mean a metric coordinate. It is necessary to reconstruct this value. Indeed, the pixelated image I PIXis a 2D image. However, the objects associated with the different Pixi pixels have different depths in the environment. For example, in Figure 3, the first Pixi pixel represents the first corner of a 300-pixel bench. The second pixel, Pix2, represents the second corner of the 300-pixel bench diametrically opposite the first Pixi pixel. This second corner lies within the depth of the image. It is therefore necessary to retrieve this information. Each depth Z of a Pixi pixel is obtained using a first artificial intelligence module capable of analyzing the pixelated image I PIX This first artificial intelligence module is pre-trained using a proprietary data sample. The AI ​​module is thus adapted to provide a depth map.

[0123]

[0082] The process in Figure 1 also includes a determination step

[0124] 105 of a danger level for said person, called danger level L of the Pixi pixel.

[0125]

[0083] As illustrated in Figure 4, this determination step 105 comprises:

[0126]

[0084] - a determination step 1051 of a semantic Si of the Pixi pixel ;

[0127]

[0085] - a determination step 1052 of the danger level L associated with said Pixi pixel.

[0128]

[0086] The danger level L of the Pixi pixel is obtained from a semantics Si of the Pixi pixel.

[0129]

[0087] By semantics Si, we mean the type of object covered by the Pixi pixel, such as a bench, a road, a bus shelter or others.

[0130]

[0088] It is thus possible to assign a danger level L to the object of the Pixi pixel. A bench or a bus shelter can have the same danger level L. A road's danger level will be higher than the danger level of the bench or bus shelter, due to an increased risk of collision with a moving vehicle. The danger levels for each Pixi pixel are assigned via a second artificial intelligence semantic segmentation module. This second module is capable of analyzing the pixelated image I PIX to give each Pixi pixel a danger level L. This second artificial intelligence module is pre-trained using a proprietary data sample.

[0131]

[0089] Alternatively, the danger level L is also defined based on the context detected in parallel. Indeed, the danger level Li is determined from the semantics of each pixel of the image produced by the AI ​​module using semantic segmentation. It may be necessary to know the general context of the location, in the same way that human cognitive analysis is able to determine the environment in which a person is located. An outdoor parking lot, an interior courtyard of a building, or a driveway of a house with an asphalt surface do not present the same danger. The invention therefore proposes adding a third classification module that determines the location (parking lot, road, interior courtyard). The danger level Li is determined according to the result of the classification. If, for example, the classification results in a parking lot environment, then the surface has a lower danger level compared to a road environment.

[0132]

[0090] The method of Figure 1 also includes a step 106 of generating a 3D point cloud. Each point Pti of the 3D point cloud is associated with a pixel Pixi of the pixelated image I PIX Furthermore, each point Pti in the point cloud is positioned within said cloud by coordinates of the point X', Y', Z. These coordinates of the point X', Y', Z, 1 are determined from the coordinates u, v, 1 of the pixel Pixi associated with said point Pti and from the depth Z of the pixel Pixi according to equation 4 below:

[0133]

[0091] (Equation 4)

[0134]

[0092] The depth Z is measured and the coordinates X' and Y' are calculated.

[0135]

[0093] The matrix [ / C -1 ] is the inverse matrix of [ ]. This matrix [ / C -1 ] is of dimension 4x3. It is calculated using linear algebra.

[0094] When A = Z, equation 4 becomes:

[0136]

[0095] (Equation 5)

[0137]

[0096] Equation 5 allows us to find the metric coordinates of the point Pti of the point cloud if the depth / (in metric system) of a point is known.

[0138]

[0097] Knowing the matrix [ / C -1 and the depth Z, it is possible to calculate the coordinates X', Y' of the point Pti from u, v.

[0139]

[0098] By applying equation 5 for each point Mû iV , it is possible to obtain the different Pti points of the point cloud in 3D.

[0140]

[0099] It is possible to assign to each point Pti a danger level L of the associated pixel Pixi. This is referred to as a 4D representation (3 dimensions of distance and the danger level L).

[0141]

[0100] The method of Figure 1 also includes a step of determining 107 a direction Dir of movement for the visually impaired person from the coordinates of all or part of the points Pti of the 3D point cloud and the danger levels Lj assigned to said points.

[0142]

[0101] As can be seen in Figure 5, determination step 107 comprises:

[0143]

[0102] - a step of eliminating 1071 points from the 3D point cloud whose height is greater than a certain value. A corrected 3D point cloud is thus formed;

[0144]

[0103] - a 1072 projection step onto the ground of the points of the corrected 3D point cloud;

[0145]

[0104] - a step of obtaining 1073 of the direction Dir of displacement from the corrected 3D point cloud.

[0146]

[0105] In one embodiment, the elimination step 1071 takes into account an inclination and / or an irregularity of the ground.

[0147]

[0106] The method of Figure 1 finally includes an audio guidance step 108 for the visually impaired person along the direction Dir of movement.

[0107] As can be seen in Figure 6, this audio guidance step 108 includes a generation step 1081 of a spatial 3D sound;

[0148]

[0108] This spatial 3D sound is a broadband sound whose source is virtually positioned spatially. Thus, the principle is to follow a sound based on the spatial perception of this sound source in 3D space and to walk in the direction of the sound.

[0149]

[0109] Audio guidance step 108 also includes:

[0150]

[0110] - a step 1082 of correction of the generated spatial 3D sound, said correction being carried out from a 3x3 rotation matrix R and a 1x3 translation matrix T;

[0151]

[0111] - a step 1083 of guiding the person independently of the rotational and translational movements of said camera.

[0152]

[0112] The rotation matrix and the translation matrix can be generated using two successive images and their associated depth maps. First, the characteristic point pairs P1 and P2 are determined using a known algorithm such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF) from the two successive images. Then, the metric positions of points P1 and P2 are extracted from the depth maps. By calculating the difference between the two positions of the same characteristic point P1 (Xi, Yi, Zi) and P2 (X2, Y2, Z2) on the three axes, the two rotation and translation vectors are determined. To minimize errors, this operation should be performed on several point pairs.

[0153]

[0113] Considering two series of 3D unit vectors as follows:

[0154]

[0114] Qi = {Pu, P 12 , P13' ■■■ ' Pin}

[0155]

[0115] Q2 = {P2I' P22' P23' —>P2n

[0156]

[0116] With P ir j, corresponding to a unit vector in Euclidean space calculated from a metric point characterized by the coordinates (X, Y, Z) whose origin is the optical axis AO. To calculate a unit vector, we divide the 3 coordinates of a metric point by the square root of the sum of the squares of its 3 coordinates.

[0157]

[0117] P u and P 2i are pairs of points corresponding to a characteristic point of the two successive images.

[0118] To obtain the angles 0 X , 0 Y and 0 Z between two unit vectors, we calculate the dot product of the projections of the two vectors onto the orthogonal plane of the desired axis.

[0158]

[0122] To calculate the translation vector, we take the differences in coordinates for each pair of points and average them over the number of points in the series.

[0159]

[0124] Alternatively, the rotation and translation matrices can be generated using a 9-axis inertial system (3 axes for the magnetometer, 3 for the gyroscope, and 3 for the accelerometer). The rotation vector is calculated using the magnetometers and the gyroscope, and the translation vector is calculated by double integration of the acceleration delivered by the accelerometer. Since the acceleration signal is very noisy, algorithms such as Extended Kalman filtering should be used to minimize the influence of noise in the calculation of the translation matrix.

[0160]

[0125] In one embodiment, the person is guided using useful information, such as a scene description or an indication of useful objects. Thus, instead of generating a sound source, a word (e.g., "bus") or a phrase (e.g., "bus stop") is generated along with its distance ("bus at 3.5 meters"), always spatially positioned in the direction of the object. During the person's movement, the information is generated in real time using 3D Text-to-Speech (TTS) technology or 3D sound. This technology involves placing the source of the speech within the space of a word or short text, such that the user perceives the voice as coming from a point in space. It is this sound source that the person must follow.

[0161]

[0126] In one embodiment, the environment around the user is described at the user's request. This environment includes obstacles and useful information. 3D TTS technology is used with metric distances.

[0127] In one embodiment, the guidance is tactile.

[0162]

[0128] In one embodiment, the guidance is haptic guidance.

[0163]

[0129] Figure 7 illustrates a system 2 for assisting the person's movement. This system 2 comprises:

[0164]

[0130] - a portable device 20 intended to be positioned on a person's head;

[0165]

[0131] - a data processing device 21.

[0166]

[0132] Figures 8 and 9 present the portable device 20 in more detail.

[0167]

[0133] This portable device 20 includes:

[0168]

[0134] - a main body 30A;

[0169]

[0135] - a first branch 30B;

[0170]

[0136] - a second branch 30C;

[0171]

[0137] - a 201 camera;

[0172]

[0138] - a first loudspeaker (not visible in figures 8 and 9) and a second loudspeaker 202;

[0173]

[0139] - a human-machine interface wheel 203.

[0174]

[0140] The main body 30A has an arc-shaped form. It is designed to allow optimal adjustment of the portable device 20 behind the user's head. This main body 20A is extended on one side by the first arm 30B and on the other side by the second arm 30C.

[0175]

[0141] The first branch 30B comprises:

[0176]

[0142] - a 301 head;

[0177]

[0143] - a 302 body;

[0178]

[0144] - an intermediate portion 303.

[0145] The head 301 is adapted to be positioned near the user's temple. It comprises a rear face 3011 arranged opposite the body 302, a front face 3012 arranged on the head 301 opposite the rear face 3011, and a lower face 3013 arranged between the front face 3012 and the rear face 3011. The lower face 3013 includes a cavity extending from the front face 3012.

[0179]

[0146] The camera 201 is located on the front face 3012. As already mentioned, this camera 201 is of the pinhole type.

[0180]

[0147] The human-machine interface wheel 203 is positioned in the cavity of the lower face 3013. This interface wheel 203 allows the portable device 20 to be controlled by the user or by voice.

[0181]

[0148] The second arm 30C also includes a head, a body, and an intermediate portion. The intermediate portion of the arm 30C includes an inner face 301 designed to make contact with the user's temple. This inner face 301 includes the second loudspeaker 202. This second loudspeaker 202 is adapted to reproduce 3D spatial sound for the user's guidance.

[0182]

[0149] It should be noted that the first branch 30B also includes the first loudspeaker (not visible in Figures 7 and 8). The first loudspeaker and the second loudspeaker 202 are thus capable of generating spatial or binaural stereophonic sound.

[0183]

[0150] In one embodiment, the wearable device 20 is in the form of glasses comprising a body and two temples extending from said body. In this configuration, each temple includes a speaker and the body includes the camera.

[0184]

[0151] As illustrated in Figure 7, the treatment device 21 is in the form of a rectangular case. Such a case can, for example, be placed in a user's pocket or in a bag carried by the user.

[0185]

[0152] Figure 10 illustrates an internal operation of the portable device 20.

[0186]

[0153] Figure 11 illustrates an internal operation of the processing device 21.

[0187]

[0154] In one embodiment, the depth Z in metric distance per pixel is determined by a first artificial intelligence module, the semantics of each pixel is determined by a second artificial intelligence module and the general classification of the environment by a third intelligence module are unified in a single artificial intelligence module.

[0188]

[0155] In a particular embodiment of the intervention, the unified artificial intelligence module generates the 2D metric ground map with pre-calculated danger levels to deal only with the orientation of the person with 3D sound.

[0189]

[0156] The proposed invention thus offers the following advantages:

[0190]

[0157] - Use of a simple image sensor for depth mapping and semantic segmentation compared to a solution using a stereoscopic system or a ToF (camera Time of Flight) system. A camera using a monocular sensor offers several advantages, such as:

[0191]

[0158] - minimal bulk;

[0192]

[0159] - a depth of mapping with constant accuracy over distance;

[0193]

[0160] - an ease of industrialization because there is no need to align the sensors;

[0194]

[0161] - there is no interpolation to adjust the size of the depth map to the RGB image or vice versa;

[0195]

[0162] - reduced energy consumption.

[0196]

[0163] The invention also provides the following advantages:

[0197]

[0164] - a construction of the environment as a metric 3D point cloud from a single image with hierarchical hazard attributes (for a pedestrian) for each point in the cloud. The primary advantage lies in the conversion of the point cloud into a ground projection of the points, which are presented in the form of a 2D matrix. Once this operation has been carried out, it is simple to perform obstacle avoidance calculations and direction angle calculations;

[0198]

[0165] - a spatial description of the environment with 3D sound composed of words or short phrases. The advantage of this solution is that it generates, in a very short time, a sensory map of the objects surrounding the user while minimizing their mental workload.

[0199]

[0166] - a correction of the movement and orientation of the head in order to generate a stationary spatial sound using two matrices (rotation and translation). In the context of the use of 3D sound, it is important to position the sound source at a fixed point when the camera is in motion characterized by two rotation vectors R and translation vectors T.

Claims

DEMANDS 1. A method for assisting the movement of a visually impaired person, said method comprising: - an acquisition step (101), by a single camera (201), of spatial data (£)) of an environment in which the visually impaired person is able to move; - a generation step (102) of a pixelated image (7 P / X ) from said spatial data (£)), said pixelated image (Z P / X ) comprising a plurality of pixels (Pixi); - for each pixel (Pixi) of the plurality of pixels: - a step of determining (103) coordinates (u, v, 1) of the pixel (Pixi); - a step of determining (104) a depth (Z) of the pixel (Pixi) by a first artificial intelligence module; - a step of determining (105) a level of danger (Li) of the pixel (Pixi) by a second artificial intelligence module; - a generation step (106) of a 3D point cloud, each point (Pti) of the point cloud being associated with a pixel (Pixi) of the pixelated image (Ipix), each point of the point cloud being positioned in the point cloud by coordinates (X', Y', Z, 1) of the point (Pti), said coordinates of the point (X 1 , Y', Z, 1) being determined from the coordinates (u, v, 1) of the pixel (Pixi) associated with said point (Pti) and from the depth (Z) of said pixel, said point (Pti) of the point cloud being affected by the danger level (Li) of said pixel (Pixi); - a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pti) of the 3D point cloud and the danger levels (Li) assigned to said points (Pti); - an audio guidance step (108) for the visually impaired person according to the determined direction (Dir) of the movement.

2. A method according to claim 1, wherein the coordinates (X', Y', Z, Y) of the point (Pti) are determined according to the following equation: an intrinsic camera matrix, such that: fx 0 Cx 0 IX] 0 fy c y 0 with fx a focal length of the camera in number of 0 o î 0. pixels along a first direction x, fy a focal length of the camera in number of pixels along a second direction y, Cx a first coordinate in pixels of the intersection of an optical axis of the camera with an image plane, Cy a second coordinate in pixels of the intersection of the optical axis of the camera with said image plane.

3. A method according to any one of claims 1 or 2, wherein the step of determining the danger level (L) of the pixel comprises a step of determining (1051) for said pixel (Pixi) a semantics (Si) of the pixel, said danger level (L) of the pixel being obtained (1052) from said semantics (Si) of the pixel (Pixi).

4. A method according to any one of claims 1 to 3, wherein, each point of the 3D point cloud having a height relative to a ground, the step of determining a direction (Dir) of displacement comprises a step of eliminating (1071) points of the 3D point cloud whose height is greater than a certain value, to form a corrected 3D point cloud.

5. Method according to claim 4, wherein said elimination step (1071) takes into account an inclination and / or an irregularity of the ground.

6. A method according to any one of claims 4 or 5, wherein the step of determining a direction of movement includes a step of projecting (1072) onto the ground the points of the corrected 3D point cloud in order to determine the direction (Dir) of movement (1073).

7. A method according to any one of claims 1 to 6, wherein the audio guidance step (108) of the visually impaired person according to the determined direction of movement includes a generation step (1081) of a spatial 3D sound.

8. Method according to claim 7, wherein the audio guidance step (108) comprises a step (1082) of correction of the generated spatial 3D sound, said correction being carried out from a 3x3 rotation matrix R and a 1x3 translation matrix T of the camera so as to guide (1083) the person independently of the rotation and translation movements of said camera.

9. A method according to any one of claims 1 to 8, wherein the determination (105) of the danger level (Li) of the pixel (Pixi) is carried out by a combination of the second artificial intelligence module with a third artificial intelligence module capable of determining a general classification of the environment.

10. A method according to any one of claims 1 to 9, wherein the first artificial intelligence module, the second artificial intelligence module and / or the third artificial intelligence module are unified into a single artificial intelligence module, referred to as the unified artificial intelligence module.

11. Method according to claim 10, wherein the unified artificial intelligence module is capable of generating a 2D metric ground map with pre-calculated hazard levels Li.

12. System for assisting the movement of a visually impaired person, said system comprising a wearable device (20) intended to be positioned on the head of said person and a processing unit (21), said wearable device (20) comprising a single camera (201) adapted to acquire spatial data (D) of an environment in which the visually impaired person is able to move, said wearable device (20) comprising an audio module (202) for audio guidance of the visually impaired person along a direction (Dir) of movement determined from the spatial data (D), said system (2) comprising at least one processor (10, 11) configured to implement: - an acquisition step (101) by the camera of spatial data (£)); - a generation step (102) of a pixelated image (I PIX ) from the spatial data (£)), said pixelated image (ZP / X ) comprising a plurality of pixels (Pixi); - for each pixel in the plurality of pixels: - a step of determining (103) the coordinates of the pixel in the pixelated image, called the coordinates (u, v, 1) of the pixel (Pixi); - a step of determining (104) a depth (Z) of the pixel (Pixi) by a first artificial intelligence module; - a step of determining (105) a level of danger (L) of the pixel (Pixi) by a second artificial intelligence module; - a generation step (106) of a 3D point cloud, each point (Pti) of the point cloud being associated with a pixel (Pixi) of the pixelated image (7 P / X ), each point of the point cloud being positioned in the point cloud by point coordinates (X', Y', Z, 1), said point coordinates (X 1 , Y', Z, 1) being determined from the coordinates (u, v, 1) of the pixel associated with said point and from the depth (Z) of said pixel, said point (Pti) of the point cloud being affected by the danger level (L) of said pixel (Pixi); - a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pti) of the 3D point cloud and the danger levels (L) assigned to said points; - an audio guidance step (108), by the audio module (202), of the visually impaired person according to the determined direction (Dir) of movement.

13. Computer program comprising software instructions which, when executed by a computer processor, implement a method for assisting the movement of a visually impaired person according to any one of claims 1 to 11.

14. A method for acquiring spatial data (£) for assisting the movement of a visually impaired person, said method comprising: - an acquisition step (101), by a single camera (201), of spatial data (£)) in which the visually impaired person is able to move; - a generation step (102) of a pixelated image (7 P / X ) from said spatial data (£)), said pixelated image (Z P / X ) comprising a plurality of pixels (Pixi); - said spatial data (£)) being intended to enable the implementation, for each pixel (Pixi) of the plurality of pixels: - a step of determining (103) coordinates (u, v, 1) of the pixel (Pixi) in the acquired image; - a step of determining (104) a depth (Z) of the pixel by a first artificial intelligence module; - a determination step (105) of a level of danger (L) for said person by a second artificial intelligence module-; - a generation step (106) of a 3D point cloud, each point (Pti) of the point cloud being associated with a pixel (Pixi) of the pixelated image (I PIX ), each point of the point cloud being positioned in the point cloud by point coordinates (X', Y', Z, ), said point coordinates (X', Y'.Z.l) being determined from the coordinates (u, v, 1) of the pixel associated with said point and from the depth of said pixel (Z), said point (Pti) of the point cloud being affected by the danger level (L) of said pixel (Pixi); - a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pti) of the 3D point cloud and the danger levels (L) assigned to said points (Pti); - said acquisition method including, in addition, an audio guidance step (108) of the visually impaired person according to the determined direction (Dir) of movement.

15. A portable device (20) intended to be positioned on the head of a visually impaired person, said portable device (20) comprising: - a single camera (201) adapted to acquire spatial data (£)) of an environment in which the visually impaired person is able to move and to generate a pixelated image (7 P / X ) from said spatial data (£)), said pixelated image (Z P / X ) comprising a plurality of pixels (Pixi); - an audio module (202) for audio guidance of the visually impaired person along a determined direction (Dir) of movement from spatial data (£)); - a processor (10) configured to implement an acquisition step (101), by the camera (201), of spatial data (£) ), said data spatial (£)) being intended to be able to be implemented by said processor (10) and / or by another processor (11), for each pixel (Pixi) of the plurality of pixels: - a step of determining (103) coordinates (u, v, 1) of the pixel (Pixi); - a step of determining (104) a depth (Z) of the pixel (Pixi) by a first artificial intelligence module; - a step of determining (105) a level of danger (L) of the pixel by a second artificial intelligence module; - a generation step (106) of a 3D point cloud, each point (Pti) of the point cloud being associated with a pixel (Pixi) of the pixelated image (Z P / X ), each point in the point cloud being positioned in the point cloud by coordinates Y', Z, 1) of the point (Pti), said coordinates (X' , Y', Z, 1) of the point (Pti) being determined from the coordinates (u, v, 1) of the pixel (Pixi) associated with said point (Pti) and from the depth (Z) of said pixel, said point (Pti) of the point cloud being affected by the danger level (L) of said pixel (Pixi); - a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pti) of the 3D point cloud and the danger levels (Lj) assigned to said points (Pti); - said processor (10) and / or said other processor (11) being configured to implement audio guidance (108) of the visually impaired person in the determined direction (Dir) of movement.

16. Computer program comprising software instructions which, when executed by a computer processor, implement a method for acquiring an image for assisting the movement of a visually impaired person according to claim 14.

17. A treatment method for assisting the mobility of a visually impaired person, said method comprising: - a generation step (106) of a 3D point cloud, each point (Pti) of the point cloud being associated with a pixel (Pixi) of a pixelated image (Ipix) obtained from spatial data (£)) via a single camera (201), each point of the point cloud being positioned in the point cloud by coordinates of the point (X', Y'.Z.l), said coordinates of the point (X', Y'.Z.l) being determined from coordinates (u, v, 1) of the pixel associated with said point and from the depth of said pixel (Z) determined by a first artificial intelligence module, said point (Pti) of the point cloud being assigned a danger level (L) of said pixel (Pixi) determined by a second artificial intelligence module, the coordinates (u, v, 1) of the pixel, the depth (Z) of the pixel, the danger level (L) of said pixel (Pixi) being obtained from said spatial data (£)); - a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pti) of the 3D point cloud and the danger levels (L) assigned to said points (Pti), said direction (Dir) determined allowing audio guidance (108) of the person.

18. Data processing device for assisting the movement of a visually impaired person, said processing device comprising a processor (11) configured to implement: - a generation step (106) of a 3D point cloud, each point (Pti) of the point cloud being associated with a pixel (Pixi) of a pixelated image (Ipix) obtained from spatial data (£) via a single camera (201), each point of the point cloud being positioned in the point cloud by point coordinates (X', Y', Z, 1), said point coordinates (X', Y', Z, 1) being determined from the coordinates (u, v, 1) of the pixel associated with said point and from the depth of said pixel (Z) determined by a first artificial intelligence module, said point (Pti) of the point cloud being assigned a danger level (L) of said pixel (Pixi) determined by a second module artificial intelligence, the coordinates (u, v, 1) of the pixel, the depth (Z) of the pixel, the danger level (L) of said pixel (Pixi) being obtained from said spatial data (£)); - a step of determining (107) a direction (Dir) of movement for the visually impaired person from the coordinates of all or part of the points (Pti) of the 3D point cloud and the danger levels (L) assigned to said points (Pti), said direction (Dir) determined allowing audio guidance (108) of the person.

19. Computer program comprising software instructions which, when executed by a computer processor, implement a data processing method for assisting the movement of a visually impaired person according to claim 17.

20. A portable device intended to be positioned on the head of a visually impaired person, said portable device being capable of implementing part of a method for assisting the movement of a visually impaired person according to any one of claims 1 to 11, said portable device comprising: - a main body (30A); - a first branch (30B); - a second branch (30C), said first branch (30B) and said second branch (30C) extending said main body (30A); - said portable device (20) comprising: - a camera (201) on the first branch (30B), said camera (201) being a pinhole camera; - a first loudspeaker on the first branch (30B) and a second loudspeaker (202) on the second branch (30C), said first loudspeaker and said second loudspeaker being capable of generating spatial 3D sound for the audio guidance for the visually impaired person along a predetermined direction (Dir) of movement; - a human-machine interface wheel (203) for controlling the portable device by the visually impaired person or by voice.

21. Portable device according to claim 20, wherein the first arm (30B) comprises a head (301), a body (302) and an intermediate part (303) between said head (301) and said body (302) and the second arm (30C) comprises a head (401), a body (402) and an intermediate part (403) between said head (401) and said body (402).

22. Portable device according to claim 21, wherein the head (301) of the first arm (30B) comprises the camera (201) and the human-machine interface wheel (203).

23. Portable device according to any one of claims 21 or 22, wherein the first loudspeaker and the second loudspeaker (202) are respectively arranged on the first arm (30B) and on the second arm (30C), opposite each other.

24. A portable device according to any one of claims 20 to 23, wherein said device is part of a system for assisting the movement of a visually impaired person according to claim 12.

Images