Systems and methods are described for analyzing video content in conjunction with historical video consumption data, and identifying and generating relationships, rules, and correlations between the video content and viewer behavior. According to one aspect, a system receives video consumption data associated with one or more output states for one or more videos. The output states generally comprise tracked and recorded viewer behaviors during videos such as pausing, rewinding, fast-forwarding, clicking on an advertisement (for Internet videos), and other similar actions. Next, the system receives metadata associated with the content of one or more videos. The metadata is associated with video content such as actors, places, objects, dialogue, etc. The system then analyzes the received video consumption data and metadata via a multivariate analysis engine to generate an output analysis of the data. The output may be a scatter plot, chart, list, or other similar type of output that is used to identify patterns associated with the metadata and the one or more output states. Finally, the system generates one or more rules incorporating the identified patterns, wherein the one or more rules define relationships between the video content (i.e. metadata) and viewer behavior (i.e. output states).