A semi-automatic 2D to 3D method with fault tolerance comprises the following steps: key frame extraction, user annotation, annotation extraction and depth conversion, super pixel segmentation, planefitting, representative pixel extraction, structural difference description, fault tolerance mechanism, energy function construction with fault-tolerant structure correlation, generalized iterative re-weighted least squares method and depth map-based drawing. The method has the following beneficial effects: a deep propagation energy model related to a fault-tolerant structure is established by means of robust estimation, and local and global constraint relationships are combined, so that the prior knowledge of a scenario can be utilized more effectively, and the quality of the estimated depthmap is enhanced while the user annotation difficulty in semi-automatic 2D to 3D conversion is reduced.