The invention discloses a color map guide-based depth map restoration and view synthesis optimization method, which comprises the following steps: firstly, detecting inconsistent regions, detecting the edge of an input depth map, performing swelling processing on the edge, and marking the swelled edge as a potential inconsistent region; secondly, constructing a weight based on an iterative reweighting-based least squares algorithm, and after weight construction is completed, performing integral solving, and updating a depth map; judging whether the iteration is performed for set times or not according to a result; and if YES, outputting the depth map, and ending the calculation, or redetecting the inconsistent regions. The color map guide-based depth map restoration and view synthesis optimization method disclosed by the invention can suppress strong noise and restore inconsistent regions between the depth map and a color map to improve the consistency between the depth map and the color map and restore a correct boundary of the depth map, and has important guiding significance for improving the quality of a synthesized view. Meanwhile, the denoising and edge retaining capacity forconsistent regions is high; and an adopted matured iterative weighted least squares model is high in adaptability to parameters, and the robustness of the model is improved.