The embodiment of the invention provides a video data processing method and device which are applied in the Internet of Videos. The method comprises the steps that after an original byte code stream of video data is received, network abstraction layer units are recognized from the original byte code stream, and whether or not the type of the network abstraction layer units is a preset type is determined; and when the type of the network abstraction layer units is the preset type, the network abstraction layer units are extracted, then the extracted network abstraction layer units are recombined, and a target byte code stream is obtained and sent to an Internet of Videos playing terminal. According to the video data processing method and device, redundant or abnormal byte code fragments canbe removed from the original byte code stream of the video data, the condition that due to the fact that a hardware decoder such as a set-top box meets the redundant or abnormal byte code fragments during decoding, the problems of blank screens, blocking, picture missing and shaking are generated when a video is played is avoided, and the fault-tolerant capacity of the decoder and the video playing smoothness are improved.