The invention provides a method for optimizing layout design rules, wherein aiming at research and development of advanced technological nodes, an initial layout is split from an initial layout, and the key graph is taken as a layout for subsequent simulation optimization; during the optimization, through combination of a light source masking and optimization simulation, a rational light source, a matched masking template and an optimized photo-etching optical model are determined; then, under the photo-etching optical model, manufacturability of the optimized light source and the optimized masking template is simulated; the design rules are optimized according to hotspot output of the manufacturability simulation; and the optimized design rules are obtained through repeated iterations. During the optimization, the key graph layout is optimized, so that feasibility and optimization efficiency are high. Through adjustment of the design rules, the quantity of iterations in the layout simulation can be reduced and the optimization efficiency can be increased on one side; and on the other side, the more optimal design rules can be obtained, design courses are simplified, and design duration is shortened.