Systems and methods for generating a digital model of a space and modifying the digital model are described. In one aspect, a
system includes one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations including: obtaining a
point cloud representation of the space, the
point cloud including multiple points; segmenting the
point cloud into: (i) an inlier point cloud including multiple inlier points, and (ii) an
outlier point cloud including multiple
outlier points, where the segmenting includes: identifying, as the inlier points, the points of the point cloud that have at least a specified likelihood of being measurements of any of multiple planes of the space; and identifying, as the
outlier points, all other points of the point cloud that are not identified as the inlier points;
processing the inlier point cloud to generate a segmented inlier point cloud that includes, for each measured plane of the space, a respective plane point cloud representing the plane; and
processing, using a neural network, the outlier point cloud to generate a segmented outlier point cloud that includes, for each of one or more objects detected in the space, a respective
object point cloud representing the object.