Robust motion from structure and structure from motion in videos

Abstract

Systems, methods, and computer program code for motion segmentation using a motion from structure approach; and related image processing tasks. Structure-from-motion systems are also described. In some implementations feature representations for pixels of an image frame are used to extrapolate from high reliability regions to lower reliability regions, based on semantic similarity as represented by the feature representations. This can link different visible parts of the same object, e.g. for generating a segmentation map for moving objects.

Description

Attorney Docket No. 45288-0580WO1ROBUST MOTION FROM STRUCTURE AND STRUCTURE FROM MOTION IN VIDEOS CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This specification claims priority to U. S. Provisional Application No. 63 / 720,749, filed November 14, 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.BACKGROUND

[0002] This specification relates to image processing using machine learning models.

[0003] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0004] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.SUMMARY

[0005] This specification generally describes systems implemented as computer programs on one or more computers in one or more locations for motion segmentation ("motion from structure'’) and related image processing tasks. This specification also describes structure-from-motion systems implemented as computer programs on one or more computers in one or more locations.

[0006] According to one aspect there is provided a method performed by one or more data processing apparatus that involves obtaining a video comprising a sequence of image frames. For one or more image frames of the sequence of image frames the method processes the image frame using an image processing neural network to generate a respective feature representation for each pixel of the image frame, and uses a trained classifier neural network to process a classifier input comprising the feature representations of the pixels of the image frame to generate a classifier output for classifying each pixel of the image frame as static or moving (dynamic).Attorney Docket No. 45288-0580WO1

[0007] In another aspect there is described a computer-implemented method that involves obtaining a video of a real-world scene captured by a moving camera, the video comprising a sequence of image frames, and processing the video using a motion segmentation system, in particular an unsupervised motion segmentation system, to generate a motion segmentation signal that identifies pixels of the video corresponding to objects that are moving in the real-world scene. The motion segmentation signal is then provided to a structure-from-motion system configured to process the video to generate an output comprising one or both of a representation of the scene and a position or pose of the moving camera. The structure-from-motion system processes the motion segmentation signal to facilitate distinguishing between objects that are moving in the real-world scene and motion parallax.

[0008] The previously described techniques can be used to provide the motion segmentation signal. In particular the classifier output can be used as the motion segmentation signal, using the pixels classified as static and moving to distinguish one or more moving objects, for attenuating artefacts from the moving objects that would otherwise arise in the structure-from-motion system.

[0009] In implementations using the motion segmentation signal to facilitate distinguishing between objects that are moving in the real-world scene and motion parallax involves using the classifier outputs for the one or more image frames to select a subset of pixels of the image frame that are predicted to be static with respect to the environment and determining the trajectory of the camera using the selected subsets of pixels of the image frames that are predicted to be static with respect to the environment.

[0010] In a further aspect there is described a computer-implemented method that involves obtaining a video comprising a sequence of image frames and, for one nor more of the sequence of image frames, processing the image frame using an image processing neural network to generate a feature representation for each pixel of the image frame, and processing the feature representation of each pixel of the image frame using a trained classifier neural network to classify the pixel as static or moving.

[0011] This can further involve processing successive pairs of image frames to determine, for each pair, an optical flow frame that represents per pixel displacement between the pair of image frames, and processing the successive pairs of image frames to determine, for each pair, a relative camera pose between the pair of image frames. The optical flowAttorney Docket No. 45288-0580WO1frame and the relative camera pose can be processed for each pair to determine, for at least one of the frames of the pair, a first set of pixels that are predicted to be static and a second set of pixels that are predicted to be moving. The classifier neural network can be trained to classify the first set of pixels as static and the second set of pixels as moving.

[0012] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0013] Implementations of the described techniques can process a video of a scene, e.g. a real-world scene captured with a moving camera, to identify one or more moving objects. More particularly, the described techniques process a sequence of 2D images, i.e. a video, to determine a segmentation map of one or more objects that are moving with respect to a background of the scene, where the segmentation map covers the moving object(s).

[0014] Implementations of the techniques can be more accurate than some prior techniques, i.e. can more accurately categorise pixels of an image / video as static or moving, particularly for real-world scenes. They are also relatively computationally efficient.

[0015] Some implementations of the described techniques combine optical flow and 3D geometric information inferred from the video (epipolar geometry), to provide improved motion segmentation information. However, even this information, whilst reliable in some regions of an image frame, can be noisy. Implementations of the described techniques can therefore also use feature representations for pixels of an image frame to extrapolate from the high reliability’ regions to lower reliability regions, based on semantic similarity (which can link different visible parts of the same object) as represented by the feature representations. The combination of these techniques is particularly effective.

[0016] In some applications the described techniques can be used to facilitate structure-from-motion image processing. In one example of structure-from-motion a 3D or 4D scene is reconstructed from a sequence of 2D images captured with a moving camera, using motion parallax. In another example of structure-from-motion the 3D position or pose of the camera is estimated. Knowing the location(s) of moving object(s) in the scene can improve the performance of such a system by enabling the system to disregard moving objects that might otherwise be mistaken for motion parallax.Attorney Docket No. 45288-0580WO1

[0017] Many conventional techniques for structure-from-motion do not work well with real-world video. Implementations of the described techniques perform structure-from-motion processing based on motion segmentation of a video, in particular as described herein, which can be a more robust approach for real-world scenes.

[0018] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG. 1 shows an example system for using a classifier neural network to separate static from dynamic parts of a video.

[0020] FIG. 2 is a flow diagram of an example method for separating static from dynamic parts of a video.

[0021] FIG. 3 is a flow diagram showing an example embodiment of the method shown in FIG. 1 also showing some optional features.

[0022] FIG. 4 is a flow diagram showing an example of a plurality of sub-steps that may form part of a step of performing a training iteration (which may, in turn, form part of the method shown in Fig. 3.

[0023] FIG. 5A shows three successive, overlaid image frames (풥t-1, 풥tand 풥t+1) of a video depicting a horse being led by a man.

[0024] FIG. 5B shows a binary mask Utof points in image frame Jt(as shown in FIG.5 A) that are likely to be dynamic.

[0025] FIG. 5C shows a binary mask Ltof points in image frame Jt(as shown in FIG.5 A) that are likely to be static.

[0026] FIG. 6 A shows an image frame (7t) of a video depicting a camel.

[0027] FIG. 6B show s a feature representation for each pixel of the image frame shown in FIG. 6A.

[0028] FIG. 6C shows a motion maskas determined by a method of the type shown in FIG. 2 and / or FIG. 3.Attorney Docket No. 45288-0580WO1

[0029] FIGS. 7A to 7D show the motion masktas determined by a method of the type shown in FIG. 2 and / or FIG. 3 after one, two, three and ten iterations respectively.

[0030] FIG. 8A show s an image frame (7t) of a video depicting a dancer.

[0031] FIG. 8B shows the motion masktas determined by a method of the type shown in FIG. 2 and / or FIG. 3 using a coarse segmentation mask.

[0032] FIG. 8C show's the motion maskas determined by a method of the type shown in FIG. 2 and / or FIG. 3 using a fine-grained video segmentation mask from SAMv2.

[0033] FIG. 9 is a flow diagram of an example process for generating an output comprising one or both of a representation of the scene and a position or pose of the moving camera from a video of a real-w orld scene captured by a moving camera.

[0034] FIG. 10 is a flow diagram of an example process for classifying each pixel of at least one image frame of a video as static or moving.

[0035] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0036] Some embodiments of the present disclosure relate to methods for robustly separating static from dynamic parts of a video. In particular, some embodiments relate to video-based motion segmentation to identify the components of a scene that are moving with respect to a fixed world frame. In turn, these methods can be used to find dynamic camera poses for the video, for example using structure-from-motion (SfM).

[0037] Figure 1 shows an example system 10 for using a classifier neural network to separate static from dynamic parts of a video. The system 10 may, for example, implement any of the methods described below with reference to Figures 2 to 10.

[0038] The system 10 comprises an image processing neural network 20 and a classifier neural network 30. The image processing neural network 20 is operable receive as input a video comprising a sequence of image framesar|d10generate a respective feature representation 22 for each pixel of one or more the image frames {fylT-i- The classifier neural network 30 is operable to a classifier input comprising the feature representations 22 of the pixels of one or more the image frames {7t}[=1and to generateAttorney Docket No. 45288-0580WO1a classifier output 32 for classifying each pixel of the image frame as static or moving (dynamic).

[0039] In some implementations the video (comprising the sequence of image frames {7t}t=1) has been captured by a camera 40 moving in an environment 50, in particular a real-world environment 50. For such embodiments, the classifier output 32 may classify each pixel of the image frame(s)as static or moving with respect to the environment 50.

[0040] Figure 2 schematically shows a method 100 according to an embodiment of the present disclosure, which may be performed by one or more data processing apparatus. The method 100 involves an initial step 102 of obtaining a video comprising a sequence of image frames.

[0041] The method 100 further comprises a step 104 of, for one or more image frames of the sequence of image frames, processing the image frame using an image processing neural network to generate a respective feature representation for each pixel of the image frame. In some embodiments, the step 104 may be carried out using the pre-trained SAMv2 video segmentation model. Details of SAMv2 may be found in Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Radle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024, which is incorporated herein by reference in its entirety.

[0042] The method 100 further comprises a step 106 of using a trained classifier neural network to process a classifier input comprising the feature representations of the pixels of the image frame to generate a classifier output for classifying each pixel of the image frame as static or moving (dynamic). In this context a pixel may be classified as being static if it corresponds to at least a part of an object in the image which is static and may be classified as being moving (dynamic) if it corresponds to at least a part of an object in the image which is moving (dynamic).

[0043] In some embodiments, given video comprising a sequence of image frames (obtained at step 102) {7t}j=1, it is a goal of the method 100 to estimate the corresponding pixel binary motion masks {t}. such that“ - 1(7“ G D), (1)Attorney Docket No. 45288-0580WO1where u are 2D pixel coordinates, D is the set of pixels which correspond to dynamic objects and the operator II (. ) is 1 if the criterion in the brackets is true and 0 if the criterion in the brackets is false.

[0044] In implementations the classifier output comprises a respective classifier score for each pixel of the image frame. The classifier scores can be used to determine a mask for segmenting the image frame into moving and static parts.

[0045] In some implementations the video has been captured by a camera moving in an environment, in particular a real-world environment, and the classifier output classifies each pixel of the image frame as static or moving with respect to the environment. That is, implementations of the method can identify pixels that correspond to one or more objects that are moving in the environment with respect to a background.

[0046] Some implementations of the method can also involve training the classifier neural network, e.g. on the or another similar video.

[0047] For example, as shown in Figure 3, this can involve a step 108 of performing a training iteration using one or more pairs of the image frames that each comprise a corresponding image frame and another image frame before or after the image frame in the sequence, e.g. using successive pairs of image frames. The training can involve comparing the pair of image frames to determine a first set of pixels of the image frame that are predicted to be static and a second set of pixels of the image frame that are predicted to be moving, and training the classifier neural network to classify the first set of pixels of the image frame as static and the second set of pixels of the image frame as moving.

[0048] The step 108 of performing a training iteration may involve identifying likely static pixels by considering optical flow between adjacent images in time, and using epipolar geometry to identify pixels in the scene whose movement can be explained solely by changes in camera pose. Note that the step 104 of generating a feature representation for each pixel of one or more image frames and the step 108 of identifying likely static pixels by considering optical flow between adjacent images in time (and using epipolar geometry to identify pixels in the scene whose movement can be explained solely by¬ changes in camera pose) may be performed in any order and / or in parallel.

[0049] The method 100 may further comprise a step 110 of using these noisy labels (determined at step 108) together with features from a pre-trained video segmentationAttorney Docket No. 45288-0580WO1model (determined at step 104) to learn a classifier that produces higher-quality and temporally stable segmentation masks.

[0050] The step 108 of performing a training iteration may be referred to as weak epipolar supervision. The step 108 of performing a training iteration may comprise a plurality of sub-steps, and an example of such sub-steps is now discussed with reference to Figure 4.

[0051] The pair of image frames can be processed to determine an optical flow frame that represents per pixel displacement between the pair of image frames. It will be appreciated that optical flow is a pattern of apparent motion of features within a scene due to relative motion between the observer (for example a camera) and the scene. The pair of image frames can also be processed to determine a relative camera pose between the pair of image frames. As discussed further below, the optical flow frame and the relative camera pose can be processed to determine the first set of pixels of the image frame that are predicted to be static and the second set of pixels of the image frame that are predicted to be moving.

[0052] There are many ways of representing optical flow. As one example an optical flow frame may comprise x-and y- displacement values for each pixel location; as another an optical flow frame may comprise x- and y- components of a vector mapping the motion of each pixel.

[0053] In implementations the relative camera pose information is used to evaluate the quality of optical flow information, to determine which of the pixel predictions (i.e. a prediction that a pixel is static and moving) is most reliable. For example, the evaluation can be based on an error, such as a re-projection error, as described below. Then the most reliable predictions can be used as the basis of the motion segmentation, e.g. by extrapolating from reliable predictions to initially less reliable predictions based on semantic similarity, e.g. as evaluated by pixel feature vectors.

[0054] In some embodiments, the step 108 of performing a training iteration may start by a step 202 of, for each of the one or more successive pairs of the image frames ]tand Jtl, computing forward and backward optical flow fields. In some embodiment, step 202 may comprise, for one or more image frames 7t, computing forward and backward optical flow fields Tt^t+andOptical flow establishes a set of dense correspondences between two frames such thatxt'xt 4" (-^t)> (2)Attorney Docket No. 45288-0580WO1where xtmay represent pixel x of an image frame at time t, x't, may represent pixel x' of an image frame at time t' and 5?t^t+1(xt) may represent a translation that the pixel x of the image frame at time t undergoes between time t and time t' due to movement of the camera during that time. In general, J,t^t+(xt) may be related to a velocity of the image due to relative motion of the camera and the scene between the pair of image frames and the time difference At = t' — t. There are many ways to determine optical flow (at step 202), e.g. RAFT (Teed and Deng, 2020), or another technique.

[0055] Optionally, to remove noisy correspondences (e.g. occlusion), correspondences that do not pass a cycle consistency (i.e.where does not return x't, to its original position xt~) may be removed from the flow fields.

[0056] The step 108 of performing a training iteration may further comprise a step 204 of, for each of the one or more successive pairs of the image frames ]tand 7t,, estimating the fundamental matrix Ftt, that represents a geometric relationship between the pixels in the two Jtand 3t, (for a static scene). For a static scene, if pairw ise correspondences (xt, x'tr) were pixel perfect, a 7-point algorithm may be employed to find the fundamental matrix Ftt / , and to estimate a relative camera pose between the two image frames Jtand Jtr. Details of a 7-point algorithm may be found in Richard Szeliski. Computer Vision: Algorithms and Applications, Springer, 2011, which is incorporated herein by reference in its entirety. To account for spurious correspondences caused by dynamic objects, the fundamental matrix Ftt, may be robustly estimated with a Median of Squares consensus measure. Details of a Median of Squares consensus measure may be found in F. R. Hampel. Beyond location parameters: Robust concepts and methods, in Bulletin of the ISI, 1975, which is incorporated herein by reference in its entirety. For example, this may be performed using RANSAC. Details of RANSAC may be found in Martin A. Fischler and Robert C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 1981, which is incorporated herein by reference in its entirety.

[0057] The step 108 of performing a training iteration may further comprise a step 206 of, for each of the one or more successive pairs of the image frames Jtand Jtl, using the fundamental matrix Ftt, (as estimated at step 204) to evaluate a quality of each flow correspondence (xt, x'tl) through epipolar geometry. For example, at step 206 theAttorney Docket No. 45288-0580WO1Sampson distance Stt,(xt) may be used as a linear approximation of the re-projection error, where the Sampson distance Stt'(xt) is given by:cf fr>=(3)where h(. ) is the homogeneous representation of a point, and d(. ) is the non-homogeneous representation. Note that, by definition, if the fundamental matrix Ftt, has been estimated correctly, if the two points (xt, x'tl) do correspond to the same point in the scene then the numerator of the right-hand side of equation (3) will be zero. Therefore, for points in image frame Jtthat are likely to be static the Sampson distance Stt> xt)~ will be smaller and for points in image frame Jtthat are likely to be moving (dynamic) the Sampson distance Stt(xt) will be larger.

[0058] The step 108 of performing a training iteration may further comprise a step 208 of, for each of the one or more successive pairs of the image framesand Jtr, using equation (3) a binary mask of points in image frame Jtthat are likely to be static (Lt) and dynamic (Ut) as:Lt= nCmaxCSt t+i. St ^i) < (4) Ut= Kmax(St-t+1, S^.i) > 0U) (5) where and 0uare two threshold values. In some embodiments, in order to account for differences in camera speed variations in scenes the two threshold values 0Land 6Umay be dependent on an average L2 norm of flow per image vt, which can be determined from the forward and / or backward optical flow fields (as determined at step 202). In one embodiment, the thresholds may be set as 0u= 2 ■ vtand 9i = 0.01 ■ vt.

[0059] As shown in the example of Figure 4, in some implementations the relative camera pose is used to determine (at step 208), for each pixel of the image frame, a respective score indicative of whether the displacement of the pixel represented by the optical flow frame can be attributed to the relative camera pose between the pair of image frames. This can involve using the relative camera pose to determine a re-projection error (for example using the Sampson distance Stt / (xt) given in equation (3)) between a location of the pixel in the image frame and a corresponding displaced location of the pixel in the image frame determined using the optical flow frame, and determining the score for the pixel based on the re-projection error. The re-projection error can be determined using aAttorney Docket No. 45288-0580WO1distance measure between the location of the pixel in the image frame and the corresponding displaced location of the pixel in the image frame.

[0060] The scores can be used to determine, for each pixel of the image frame, whether the pixel belongs to the first set of pixels of the image frame that are predicted to be static, and / or whether the pixel belongs to the second set of pixels of the image frame that are predicted to be moving. As discussed above, this can involve determining whether a pixel belongs to the first set of pixels of the image frame by comparing the score for the pixel with a first thresholdsee equation (4)) and determining whether the pixel belongs to the second set of pixels of the image frame by comparing the score for the pixel with a second threshold (0U, see equation (5)).

[0061] In a particular example, a first image is combined with a determined optical flow frame to obtain a prediction of a second image. Then each pixel of the first image frame can have a corresponding displaced location in the second image frame determined using the optical flow frame. The first and second images can then be processed (for example at step 204) to determine the so-called “fundamental matrix” Ftt, (in the field of epipolar geometry). This is a matrix that encodes the intrinsic projective or epipolar geometry between two views, i.e. between the two images (and which does not depend on the camera’s intrinsic parameters or on the scene). There are various ways to determine the fundamental matrix Ftt,; in the described techniques it can be useful to use a best fit method for robustness.

[0062] The quality of the pixel classifications as static or moving can then be evaluated using the fundamental matrix Ftt,, e.g. by evaluating the quality of the optical flow-determined correspondence between each pixel of the first image frame and the corresponding displaced location in the second image frame (e g. using a homogeneous representation of points / pixels). For example, pixels can be determined as reliably static or moving if they have likelihoods defined by respective first and second thresholds (see equations (4) and (5) above). For example, if the re-projection error is less than the first threshold a pixel / point can be reliably determined as static; and if the re-proj ection error is greater than the second threshold a pixel / point can be reliably determined as moving. Optionally the maximum re-projection error for adjacent (in time) pairs of frames can be used.Attorney Docket No. 45288-0580WO1

[0063] An example of the masks Utor Lt(determined at step 208) is shown in Figures 5A to 5C. Figure 5A shows three successive, overlaid image frames (7t-i, 7tand 7t+1) of a video depicting a horse being led by a man. Figure 5B shows a binary mask Utof points in image frame Jtthat are likely to be dynamic and Figure 5C shows a binary mask Ltof points in image frame Jtthat are likely to be static. Note that in binary mask Utthe second set of pixels of the image frame that are predicted to be dynamic are highlighted and in binary mask Ltthe first set of pixels of the image frame that are predicted to be static are highlighted. It can be seen from a comparison of Figures 5A and 5B that, in general, some pixels of the image frame do not belong to the first set or the second set. That is, the masks Utor Ltdetermined from equations (4) and (5) each provide a sparse and noisy per image-pair representation of the static and dynamic decomposition of a scene respectively.

[0064] It may be an objective of the method 100 shown in Figures 2 and 3 to produce a dense, robust and temporally consistent pixel-by-pixel estimate of “. To achieve this, some embodiments of the method 100 take advantage of a well-behaved feature space of a video segmentation model (as determined at step 104, for example using a pre-trained SAMv2 video segmentation model). The intuition is that pixels corresponding to the same object in a video should be close in feature space, as these models are specifically trained for object-level segmentation. Therefore, in some embodiments, within this feature space, a lightweight classifier is supervised with labels given by equations (4) and (5) above, as now discussed.

[0065] In particular, at step 110, given an image frame 7t, a corresponding feature map Gtcan be extracted (at step 204), for example from e.g. the last layer of a SAMv2 or other video segmentation model encoder and a classifier, e.g. a shallow multi-layer perceptron (MLP), J-Cemay be trained to classify the feature space using the loss:= x(max(Ut- (Gt), 0) + K(max(e(Gt) - Lt, 0), (6) where Ltis the binary complement of Ltand the use of max ensures that Jf is only supervised at the pixel locations activated in Utor Lt. Optionally, a Geman-McClure robust kernel K with a fixed temperature may be added to make the solution more robust to imperfections to the automatic selection of 0Uand 0; across sequences. In general, a robust kernel is one that gives a relatively reduced weight to outliers. Further relevant details of such a loss function may be found in Sara Sabour, Suhani Vora, DanielAttorney Docket No. 45288-0580WO1Duckworth, Ivan Krasin, David J Fleet, and Andrea Tagliasacchi. Robustnerf: Ignoring distractors with robust losses, in CVPR, 2023, which is incorporated herein by reference in its entirety.

[0066] Optionally, the classifier (MLP) weights may be regularized, e.g. using a Lipschitz or other regularizer Lreg. The classifier can be trained across all T time steps t within a video sequence:£(0) = ^4M0) + W0)-(7)t

[0067] A Lipschitz regularizer Lregis generally a regularizer with a smoothness constraint; an example is described in Hsueh-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany, "‘Learning smooth neural functions via lipschitz regularization”, in SIGGRAPH, 2022, which is incorporated herein by reference in its entirety.

[0068] Once trained, motion masks are found (for example at step 106) by thresholding, e.g. as:= n(Jf0(Gt) > 0.5). (8)

[0069] Figure 6A shows an image frame (ty) of a video depicting a camel. Figure 6B shows a feature representation for each pixel of the image frame shown in Figure 6 A (for example as may be generated at step 104). These features are leveraged by the method 100 shown in Figure 3 to train the classifier on the sparse and noisy labels Utand Ltfrom epipolar supervision to generate a coherent motion maskt. Figure 6C shows the motion maskas defined by equation (8) (and determined at step 106).

[0070] In some videos extreme situations can occur, where foreground objects completely occlude the scene. This results in highly unreliable estimates of the fundamental matrix Ftt„ and therefore low-quality pseudo-labels Utand Ltfor supervising. Therefore, in some embodiments, frames may be disregarded if less than 50% of pixels are marked as inliers so as to boost performance of the method 100.

[0071] As discussed above, in some implementations training the classifier neural network involves processing the image frame using the image processing neural network to generate a respective feature representation for each pixel of the image frame (for example at step 104), and using the classifier neural network to process a classifier training input comprising the feature representations of the pixels of the image frame toAttorney Docket No. 45288-0580WO1generate, in accordance with current values of trainable parameters, e g. weights, of the classifier network, respective classifier scores for the pixels of the image frame.

[0072] As in the example described above with reference to equation (6), the current values of trainable parameters of the classifier network can be adjusted to optimize an objective function £gLipcomprising (i) a first term (the second term on the right-hand side of equation (6)) comparing, for each pixel, the respective classifier score for the pixel with a value for the pixel obtained by applying a binary mask to the first set of pixels, and (ii) a second term (the first term on the right-hand side of equation (6)) comparing, for each pixel, the respective classifier score for the pixel with a value for the pixel obtained by applying a binary7mask to the second set of pixels. The comparing may be implemented using, e.g. a difference (as in equation (6)) or ratio.

[0073] In general, the classifier can be trained by backpropagating gradients of the objective function or loss to update learnable parameters, e.g. weights, of the classifier. This may use any appropriate gradient descent optimization algorithm, e.g. Adam or another optimization algorithm.

[0074] In some implementations (as in equation (6)) the objective function applies a robust kernel to one or both of the first term and the second term. In general, a robust kernel is one that gives a relatively reduced weight to outliers.

[0075] In some implementations the objective function comprises respective first and second terms for only a proper subset (i.e. less than all) of the image frames in the sequence. Each image frame can be selected for inclusion in the proper subset if a number of pixels of the image frame belonging to one or both of the first set of pixels that are predicted to be static for the image frame or the second set of pixels that are predicted to be moving for the image frame exceeds a threshold. For example, as described above, in some embodiments, frames may be disregarded if less than 50% of pixels are marked as inliers so as to boost performance of the method 100.

[0076] Once the classifier J~Cghas been trained, the predicted masks better approximate the moving objects than Utand Ltas Jfgalso considers semantic appearance, rather than just epipolar geometry. Therefore, as shown in Figure 4, the method 100 may comprise an iterative refinement process 112. The iterative refinement process 112 may entail the use of J~Cgto remove bad correspondences, re-computing (for example by repeating step 108) the fundamental matrices Ftt,, yielding improved masks Utand Lt,Attorney Docket No. 45288-0580WO1and using the updated masks (for example by repeating step 110) to fine-tune J-C8from the previous step (i.e. from a previous implementation of step 110).

[0077] Therefore, optionally an iterative refinement process 112 can be used. This can involve, after performing the training iteration (at step 110), performing one or more further training iterations (for example by repeating steps 108 and 110). Each further training iteration can involve one or more pairs of the image frames that each comprises a corresponding image frame and another of the image frames before or after the image frame in the sequence. The image frame can be processed using the image processing neural network to generate a respective feature representation for each pixel of the image frame. The classifier neural network can be used to process the classifier input comprising the feature representations of the pixels of the image frame to generate the classifier output for classifying each pixel of the image frame as static or moving. The classifier output can be used to select a proper subset (less than all) of the pixels of the image frame that can be reliably determined to be, e.g., (i) static, or (ii) moving, or optionally, either static or moving. Such a reliable determination may be, e.g. a determination with greater than a predetermined likelihood, or a determination that is relatively of greater than likelihood than for pixels not in the proper subset.

[0078] The pair of image frames can be compared to determine a first set of pixels of the image frame that are predicted to be static and a second set of pixels of the image frame that are predicted to be moving. Either or both of the first and second sets of pixels can be filtered to include only pixels that belong to the proper subset of the pixels of the image frame that can be clearly determined to be either static or moving. After the filtering the classifier neural network can be further trained to classify the first set of pixels of the image frame as static and the second set of pixels of the image frame as moving.

[0079] In some implementations the (trained) classifier neural network comprises (or consists of) a (trained) multilayer perceptron. In some implementations the image processing neural network has been trained to perform one or more of an image segmentation task (e.g. it may comprise a Segment Anything Model) and a point tracking task (e.g. it may comprise a TAPIR Model - Doersch et al., arXiv:2306.08637, 2023). In some implementations, the image processing task has been trained to perform a task (e.g. a dense estimation task) that predicts a respective classification for each of the pixels in the image frame.Attorney Docket No. 45288-0580WO1

[0080] It has been found that there is typically an optimal number of iterations after which performance saturates. This is demonstrated in Figures 7A to 7D, which show the motion masktas defined by equation (8) (and determined at step 106) after one, two, three and ten iterations respectively. In this example it was found that two iterations was optimal.

[0081] As the feature maps Gt(which may be output at step 104) are of typically relatively low-resolution, the segmentation masks may be initially coarse. However, some video segmentation models, such as SAMv2, allow for coarse masks, points and boxes to be specified as input for any frame, and can output temporally consistent fine-grained video segmentation masks. In some embodiments, the method 100 may exploit this capability', so as to convert coarse masks from to higher resolution masks. The benefit of this is demonstrated in Figures 8A to 8C.

[0082] Figure 8A shows an image frame (7t) of a video depicting a dancer. Figure 8B shows the motion masktas defined by equation (8) (and determined at step 106) without using this capability from (in this example) SAMv2 whereas Figure 8C shows the motion masktas defined by equation (8) (and determined at step 106) using this capability’ from S AMv2. In particular, it can be seen from a comparison of Figures 8B and 8C that by using this capability from S AMv2 the finer details around the fingers and the dress frills of the dancer are more accurately reflected by the motion mask Mt.

[0083] The novel motion segmentation method 100 shown in Figures 2 and 3 and as described above has been developed with a goal of improving structure from motion for in-the-wild video. It has been found that this method 100 substantially outperforms existing unsupervised motion segmentation techniques on standard benchmarks. The method 100 can use off-the-shelf optical flow and semantic features software.

[0084] Figure 9 schematically shows a computer-implemented method 300 according to an embodiment of the present disclosure. The method 300 shown in Figure 9 comprises a step 302 of obtaining a video of a real-world scene captured by a moving camera, the video comprising a sequence of image frames. The method 300 shown in Figure 9 further comprises a step 304 of processing the video using a motion segmentation system, in particular an unsupervised motion segmentation system, to generate a motion segmentation signal that identifies pixels of the video corresponding to objects that are moving in the real-world scene. The method 300 show n in Figure 9 comprises a step 306Attorney Docket No. 45288-0580WO1wherein the motion segmentation signal is provided to a structure-from-motion system configured to process the video to generate an output comprising one or both of a representation of the scene and a position or pose of the moving camera. The structure-from-motion system processes the motion segmentation signal to facilitate distinguishing between objects that are moving in the real-world scene and motion parallax.

[0085] The previously described techniques can be used to provide the motion segmentation signal. In particular the classifier output (for example at step 106 of the method 100 shown in Figures 2 and 3) can be used as the motion segmentation signal, using the pixels classified as static and moving to distinguish one or more moving objects, for attenuating artefacts from the moving objects that would otherwise arise in the structure-from-motion system.

[0086] In implementations using the motion segmentation signal to facilitate distinguishing between objects that are moving in the real-world scene and motion parallax involves using the classifier outputs for the one or more image frames to select a subset of pixels of the image frame that are predicted to be static with respect to the environment and determining the trajectory of the camera using the selected subsets of pixels of the image frames that are predicted to be static with respect to the environment.

[0087] Figure 10 schematically shows a computer-implemented method 400 according to an aspect of the present disclosure. The method 400 comprises a step 402 of obtaining a video comprising a sequence of image frames. The method 400 further comprises a step 404 of, for one or more of the sequence of image frames, processing the image frame using an image processing neural network to generate a feature representation for each pixel of the image frame. The method 400 further comprises a step 406 of, for one or more of the sequence of image frames, processing the feature representation of each pixel of the image frame using a trained classifier neural network to classify the pixel as static or moving.

[0088] This method 400 can further involve processing successive pairs of image frames to determine, for each pair, an optical flow frame that represents per pixel displacement between the pair of image frames, and processing the successive pairs of image frames to determine, for each pair, a relative camera pose between the pair of image frames. The optical flow frame and the relative camera pose can be processed for each pair to determine, for one of the frames of the pair, a first set of pixels that are predicted to beAttorney Docket No. 45288-0580WO1static and a second set of pixels that are predicted to be moving. The classifier neural network can be trained to classify the first set of pixels as static and the second set of pixels as moving.

[0089] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary7software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0090] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry7, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry7information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0091] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used forAttorney Docket No. 45288-0580WO1this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0092] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.Attorney Docket No. 45288-0580WO1

[0093] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of Al and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0094] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in Al and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or applicationspecific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0095] Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machineAttorney Docket No. 45288-0580WO1learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The essential elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the Al model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs. or flash memory for persistent data storage.

[0096] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory7devices such as read-only memory7(ROM), solid-state drives (SSDs), and flash memory devices: hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability7or permanence.

[0097] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory7, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms.Attorney Docket No. 45288-0580WO1The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0098] Machine learning models can be implemented and deployed using machine learning frameworks, such as PyTorch. TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0099] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0100] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system.

[0101] In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can thenAttorney Docket No. 45288-0580WO1process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0102] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0103] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0104] This specification also provides the subject-matter of the following numbered clauses:

[0105] 1. A method performed by one or more data processing apparatus, the method comprising:obtaining a video comprising a sequence of image frames; andfor one or more image frames of the sequence of image frames:Attorney Docket No. 45288-0580WO1processing the image frame using an image processing neural network to generate a respective feature representation for each pixel of the image frame; and using a trained classifier neural network to process a classifier input comprising the feature representations of the pixels of the image frame to generate a classifier output for classifying each pixel of the image frame as static or moving.

[0106] 2. The method of clause 1, wherein the video has been captured by a camera moving in an environment and the classifier output is for classifying each pixel of the image frame as static or moving with respect to the environment.

[0107] 3. The method of clause 1 or 2, further comprising, performing a training iteration, comprising, for one or more pairs of the image frames that each comprise a corresponding image frame and another image frame before or after the image frame in the sequence:comparing the pair of image frames to determine a first set of pixels of the image frame that are predicted to be static and a second set of pixels of the image frame that are predicted to be moving; andtraining the classifier neural network to classify the first set of pixels of the image frame as static and the second set of pixels of the image frame as moving.

[0108] 4. The method of clause 3, wherein comparing the pair of image frames comprising the image frame and another of the image frames of the sequence to determine the first set of pixels of the image frame that are predicted to be static and the second set of pixels of the image frame that are predicted to be moving comprises:processing the pair of image frames to determine an optical flow frame that represents per pixel displacement between the pair of image frames;processing the pair of image frames to determine a relative camera pose between the pair of image frames; andprocessing the optical flow frame and the relative camera pose to determine the first set of pixels of the image frame that are predicted to be static and the second set of pixels of the image frame that are predicted to be moving.

[0109] 5. The method of clause 4, wherein processing the optical flow frame and the relative camera pose to determine the first set of pixels of the image frame that are predicted to be static and the second set of pixels of the image frame that are predicted to be moving comprises:Attorney Docket No. 45288-0580WO1using the relative camera pose to determine, for each pixel of the image frame, a respective score indicative of whether the displacement of the pixel represented by the optical flow frame can be attributed to the relative camera pose between the pair of image frames; andusing the scores to determine, for each pixel of the image frame, (i) whether the pixel belongs to the first set of pixels of the image frame that are predicted to be static, and (ii) whether the pixel belongs to the second set of pixels of the image frame that are predicted to be moving.

[0110] 6. The method of clause 5, wherein using the relative camera pose to determine, for each pixel of the image frame, a respective score indicative of whether the displacement of the pixel represented by the corresponding pixel of the optical flow frame can be attributed to the relative camera pose between the pair of image frames comprises:using the relative camera pose to determine a re-projection error between a location of the pixel in the image frame and a corresponding displaced location of the pixel in the image frame determined using the optical flow frame; anddetermining the score for the pixel based on the re-projection error.

[0111] 7. The method of clause 6, wherein the re-projection error is determined using a distance measure between the location of the pixel in the image frame and the corresponding displaced location of the pixel in the image frame.

[0112] 8. The method of any one of clauses 5-7, wherein using the scores to determine, for each pixel of the image frame, (i) whether the pixel belongs to the first set of pixels of the image frame that are predicted to be static; and (ii) whether the pixel belongs to the second set of pixels of the image frame that are predicted to be moving comprises:determining whether the pixel belongs to the first set of pixels of the image frame by comparing the score for the pixel with a first threshold; anddetermining whether the pixel belongs to the second set of pixels of the image frame by comparing the score for the pixel with a second threshold.

[0113] 9. The method of any one of clauses 3-8, wherein the classifier output comprises a respective classifier score for each pixel.

[0114] 10. The method of clause 9, further comprising using the classifier scores to determine a mask for segmenting the image frame into moving and static parts.Attorney Docket No. 45288-0580WO1

[0115] 11. The method of clause 9 or 10, wherein training the classifier neural network to classify the first set of pixels of the image frame as static and the second set of pixels of the image frame as moving comprises:processing the image frame using the image processing neural network to generate a respective feature representation for each pixel of the image frame; andusing the classifier neural network to process a classifier training input comprising the feature representations of the pixels of the image frame to generate, in accordance with current values of parameters of the classifier network, respective classifier scores for the pixels of the image frame;adjusting the current values of parameters of the classifier network to optimize an objective function comprising (i) a first term comparing, for each pixel, the respective classifier score for the pixel with a value for the pixel obtained by applying a binary mask to the first set of pixels, and (ii) a second term comparing, for each pixel, the respective classifier score for the pixel with a value for the pixel obtained by applying a binary mask to the second set of pixels.

[0116] 12. The method of clause 11, wherein the objective function applies a robust kernel to one or both of the first term and the second term.

[0117] 13. The method of clause 11 or 12, wherein the objective function comprises respective first and second terms for only a proper subset of the image frames in the sequence, each image frame being selected for inclusion in the proper subset if a number of pixels of the image frame belonging to one or both of the first set of pixels that are predicted to be static for the image frame or the second set of pixels that are predicted to be moving for the image frame exceeds a threshold.

[0118] 14. The method of any one of clauses 3-13, further comprising, after performing the training iteration, performing one or more further training iterations, each further training iteration comprising, for one or more pairs of the image frames that each comprises a corresponding image frame and another of the image frames before or after the image frame in the sequence:processing the image frame using the image processing neural network to generate a respective feature representation for each pixel of the image frame; andusing the classifier neural network to process the classifier input comprising the feature representations of the pixels of the image frame to generate, in accordance withAttorney Docket No. 45288-0580WO1current values of parameters of the classifier network, the classifier output for classifying each pixel of the image frame as static or moving;using the classifier output to select a proper subset of the pixels of the image frame that can be reliably determined to be (i) static, or (ii) moving;comparing the pair of image frames to determine a first set of pixels of the image frame that are predicted to be static and a second set of pixels of the image frame that are predicted to be moving;filtering either or both of the first and second sets of pixels to include only pixels that belong to the proper subset of the pixels of the image frame that can be clearly determined to be either static or moving; andafter the filtering, further training the classifier neural network to classify the first set of pixels of the image frame as static and the second set of pixels of the image frame as moving.

[0119] 15. The method of any one of the preceding clauses, wherein using the trained classifier neural network to process a classifier input comprising the feature representations of the pixels of the image frame to generate a classifier output for classifying each pixel of the image frame as static or moving comprises:using a trained multilayer perceptron to process a classifier input comprising the feature representations of the pixels of the image frame to generate a classifier output for classifying each pixel of the image frame as static or moving.

[0120] 16. The method of any one of the preceding clauses, wherein the image processing neural network has been trained to perform one or more of: an image segmentation task and a point tracking task.

[0121] 17. A method performed by one or more data processing apparatus, the method comprising:obtaining a video of a real-world scene captured by a moving camera, the video comprising a sequence of image frames;processing the video using a motion segmentation system to generate a motion segmentation signal that identifies pixels of the video corresponding to objects that are moving in the real-world scene; andproviding the motion segmentation signal to a structure-from-motion system configured to process the video to generate an output comprising one or both of aAttorney Docket No. 45288-0580WO1representation of the scene and a position or pose of the moving camera, the structure-from-motion system processing the motion segmentation signal to facilitate distinguishing between objects that are moving in the real-world scene and motion parallax.

[0122] 18. The method of clause 17, wherein processing the video using the motion segmentation system to generate the motion segmentation signal comprises performing the method of any one of clauses 1-16 and using the classifier output as the motion segmentation signal.

[0123] 19. The method of clause 18, wherein processing the motion segmentation signal to facilitate distinguishing between objects that are moving in the real -world scene and motion parallax comprises:using the classifier outputs for the one or more image frames to select a subset of pixels of the image frame that are predicted to be static with respect to the environment; anddetermining the trajectory of the camera using the selected subsets of pixels of the image frames that are predicted to be static with respect to the environment.

[0124] 20. A method performed by one or more data processing apparatus for determining a trajectory of a camera, the method comprising:performing the method of any one of clauses 1-16, wherein the video is obtained using a camera moving within an environment; andusing the classifier outputs for the one or more image frames to select a subset of pixels of the image frame that are predicted to be static with respect to the environment; anddetermining the trajectory of the camera using the selected subsets of pixels of the image frames that are predicted to be static with respect to the environment.

[0125] 21. A computer-implemented method comprising:obtaining a video comprising a sequence of image frames; and, for one nor more of the sequence of image frames:processing the image frame using an image processing neural network to generate a feature representation for each pixel of the image frame; andprocessing the feature representation of each pixel of the image frame using a trained classifier neural network to classify the pixel as static or moving.Attorney Docket No. 45288-0580WO1

[0126] 22. The method of clause 21, further comprising:processing successive pairs of image frames to determine, for each pair, an optical flow frame that represents per pixel displacement between the pair of image frames; processing the successive pairs of image frames to determine, for each pair, a relative camera pose between the pair of image frames; and for each pair:processing the optical flow frame and the relative camera pose to determine, for one of the frames of the pair, a first set of pixels that are predicted to be static and a second set of pixels that are predicted to be moving; andtraining the classifier neural network to classify the first set of pixels as static and the second set of pixels as moving.

[0127] 23. A system comprising:one or more computers; andone or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of clauses 1-22.

[0128] One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any one of clauses 1-22.

[0129] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0130] What is claimed is:

Claims

Attorney Docket No. 45288-0580WO1CLAIMS1. A method performed by one or more data processing apparatus, the method comprising:obtaining a video comprising a sequence of image frames; andfor one or more image frames of the sequence of image frames:processing the image frame using an image processing neural network to generate a respective feature representation for each pixel of the image frame; and using a trained classifier neural network to process a classifier input comprising the feature representations of the pixels of the image frame to generate a classifier output for classifying each pixel of the image frame as static or moving.

2. The method of claim 1. wherein the video has been captured by a camera moving in an environment and the classifier output is for classifying each pixel of the image frame as static or moving with respect to the environment.

3. The method of claim 1 or 2, further comprising, performing a training iteration, comprising, for one or more pairs of the image frames that each comprise a corresponding image frame and another image frame before or after the image frame in the sequence:comparing the pair of image frames to determine a first set of pixels of the image frame that are predicted to be static and a second set of pixels of the image frame that are predicted to be moving; andtraining the classifier neural network to classify the first set of pixels of the image frame as static and the second set of pixels of the image frame as moving.

4. The method of claim 3, wherein comparing the pair of image frames comprising the image frame and another of the image frames of the sequence to determine the first set of pixels of the image frame that are predicted to be static and the second set of pixels of the image frame that are predicted to be moving comprises:processing the pair of image frames to determine an optical flow frame that represents per pixel displacement between the pair of image frames;Attorney Docket No. 45288-0580WO1processing the pair of image frames to determine a relative camera pose between the pair of image frames; andprocessing the optical flow frame and the relative camera pose to determine the first set of pixels of the image frame that are predicted to be static and the second set of pixels of the image frame that are predicted to be moving.

5. The method of claim 4, wherein processing the optical flow frame and the relative camera pose to determine the first set of pixels of the image frame that are predicted to be static and the second set of pixels of the image frame that are predicted to be moving comprises:using the relative camera pose to determine, for each pixel of the image frame, a respective score indicative of whether the displacement of the pixel represented by the optical flow frame can be attributed to the relative camera pose between the pair of image frames; andusing the scores to determine, for each pixel of the image frame, (i) whether the pixel belongs to the first set of pixels of the image frame that are predicted to be static, and (ii) whether the pixel belongs to the second set of pixels of the image frame that are predicted to be moving.

6. The method of claim 5, wherein using the relative camera pose to determine, for each pixel of the image frame, a respective score indicative of whether the displacement of the pixel represented by the corresponding pixel of the optical flow frame can be attributed to the relative camera pose between the pair of image frames comprises: using the relative camera pose to determine a re-projection error between a location of the pixel in the image frame and a corresponding displaced location of the pixel in the image frame determined using the optical flow frame; anddetermining the score for the pixel based on the re-projection error.

7. The method of claim 6, wherein the re-projection error is determined using a distance measure between the location of the pixel in the image frame and the corresponding displaced location of the pixel in the image frame.Attorney Docket No. 45288-0580WO18. The method of any one of claims 5-7, wherein using the scores to determine, for each pixel of the image frame, (i) whether the pixel belongs to the first set of pixels of the image frame that are predicted to be static; and (ii) whether the pixel belongs to the second set of pixels of the image frame that are predicted to be moving comprises:determining whether the pixel belongs to the first set of pixels of the image frame by comparing the score for the pixel with a first threshold; anddetermining whether the pixel belongs to the second set of pixels of the image frame by comparing the score for the pixel with a second threshold.

9. The method of any one of claims 3-8, wherein the classifier output comprises a respective classifier score for each pixel.

10. The method of claim 9, further comprising using the classifier scores to determine a mask for segmenting the image frame into moving and static parts.

11. The method of claim 9 or 10, wherein training the classifier neural network to classify the first set of pixels of the image frame as static and the second set of pixels of the image frame as moving comprises:processing the image frame using the image processing neural network to generate a respective feature representation for each pixel of the image frame; andusing the classifier neural network to process a classifier training input comprising the feature representations of the pixels of the image frame to generate, in accordance with current values of parameters of the classifier network, respective classifier scores for the pixels of the image frame;adjusting the current values of parameters of the classifier netw ork to optimize an objective function comprising (i) a first term comparing, for each pixel, the respective classifier score for the pixel with a value for the pixel obtained by applying a binary mask to the first set of pixels, and (ii) a second term comparing, for each pixel, the respective classifier score for the pixel with a value for the pixel obtained by applying a binary mask to the second set of pixels.Attorney Docket No. 45288-0580WO112. The method of claim 11, wherein the objective function applies a robust kernel to one or both of the first term and the second term.

13. The method of claim 11 or 12, wherein the objective function comprises respective first and second terms for only a proper subset of the image frames in the sequence, each image frame being selected for inclusion in the proper subset if a number of pixels of the image frame belonging to one or both of the first set of pixels that are predicted to be static for the image frame or the second set of pixels that are predicted to be moving for the image frame exceeds a threshold.

14. The method of any one of claims 3-13, further comprising, after performing the training iteration, performing one or more further training iterations, each further training iteration comprising, for one or more pairs of the image frames that each comprises a corresponding image frame and another of the image frames before or after the image frame in the sequence:processing the image frame using the image processing neural network to generate a respective feature representation for each pixel of the image frame; andusing the classifier neural network to process the classifier input comprising the feature representations of the pixels of the image frame to generate, in accordance with current values of parameters of the classifier network, the classifier output for classifying each pixel of the image frame as static or moving;using the classifier output to select a proper subset of the pixels of the image frame that can be reliably determined to be (i) static, or (ii) moving;comparing the pair of image frames to determine a first set of pixels of the image frame that are predicted to be static and a second set of pixels of the image frame that are predicted to be moving;filtering either or both of the first and second sets of pixels to include only pixels that belong to the proper subset of the pixels of the image frame that can be clearly determined to be either static or moving; andafter the filtering, further training the classifier neural network to classify the first set of pixels of the image frame as static and the second set of pixels of the image frame as moving.Attorney Docket No. 45288-0580WO115. The method of any one of the preceding claims, wherein using the trained classifier neural network to process a classifier input comprising the feature representations of the pixels of the image frame to generate a classifier output for classifying each pixel of the image frame as static or moving comprises:using a trained multilayer perceptron to process a classifier input comprising the feature representations of the pixels of the image frame to generate a classifier output for classifying each pixel of the image frame as static or moving.

16. The method of any one of the preceding claims, wherein the image processing neural network has been trained to perform one or more of: an image segmentation task and a point tracking task.

17. A method performed by one or more data processing apparatus, the method comprising:obtaining a video of a real-world scene captured by a moving camera, the video comprising a sequence of image frames;processing the video using a motion segmentation system to generate a motion segmentation signal that identifies pixels of the video corresponding to objects that are moving in the real-world scene; andproviding the motion segmentation signal to a structure-from-motion system configured to process the video to generate an output comprising one or both of a representation of the scene and a position or pose of the moving camera, the structure-from-motion system processing the motion segmentation signal to facilitate distinguishing between objects that are moving in the real-world scene and motion parallax.

18. The method of claim 17, wherein processing the video using the motion segmentation system to generate the motion segmentation signal comprises performing the method of any one of claims 1-16 and using the classifier output as the motion segmentation signal.Attorney Docket No. 45288-0580WO119. The method of claim 18, wherein processing the motion segmentation signal to facilitate distinguishing between objects that are moving in the real-world scene and motion parallax comprises:using the classifier outputs for the one or more image frames to select a subset of pixels of the image frame that are predicted to be static with respect to the environment; anddetermining the trajectory of the camera using the selected subsets of pixels of the image frames that are predicted to be static with respect to the environment.

20. A method performed by one or more data processing apparatus for determining a trajectory' of a camera, the method comprising:performing the method of any one of claims 1-16, wherein the video is obtained using a camera moving within an environment; andusing the classifier outputs for the one or more image frames to select a subset of pixels of the image frame that are predicted to be static with respect to the environment; anddetermining the trajectory’ of the camera using the selected subsets of pixels of the image frames that are predicted to be static with respect to the environment.

21. A computer-implemented method comprising:obtaining a video comprising a sequence of image frames; and, for one nor more of the sequence of image frames:processing the image frame using an image processing neural network to generate a feature representation for each pixel of the image frame; andprocessing the feature representation of each pixel of the image frame using a trained classifier neural network to classify the pixel as static or moving.

22. The method of claim 21, further comprising:processing successive pairs of image frames to determine, for each pair, an optical flow frame that represents per pixel displacement between the pair of image frames; processing the successive pairs of image frames to determine, for each pair, a relative camera pose between the pair of image frames; and for each pair:Attorney Docket No. 45288-0580WO1processing the optical flow frame and the relative camera pose to determine, for one of the frames of the pair, a first set of pixels that are predicted to be static and a second set of pixels that are predicted to be moving; andtraining the classifier neural network to classify the first set of pixels as static and the second set of pixels as moving.

23. A system comprising:one or more computers; andone or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-22.

24. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any one of claims 1-22.

Images