The invention provides an optimized video data coding method based on boundary adaptive transform. The method mainly comprises steps: according to the existing DCT (Discrete Cosine Transform) matrix for video coding, a new transform matrix is generated, the new transform matrix and the existing DST (Discrete Sine Transform)/DCT matrix are combined into a new transform matrix set; according to a prediction mode, the size of a transform unit, and the relationship between the boundary information of a transform unit and the boundary information of a prediction unit, a group of transform matrixes is selected from the new transform matrix set; and by using the group of new transform matrixes, inputted video data are transformed, and a coding stream is generated. The method of the invention makes full use of distribution characteristics of residual matrixes after inter-frame prediction, and based on the boundary information of the prediction unit and the transform unit, a different transform matrix is adopted for a different boundary characteristic to carry out transform. The optimized method can select a more efficient transform matrix for transform in a targeted mode according to the characteristics of residuals after inter-frame prediction, and the coding efficiency is thus improved.