An equalizer control system is provided. The equalizer control system includes a first sound mode identifier to identify a first sound mode for controlling the equalizer from a plurality of sound modes, by classifying an audio feature value, extracted from a sequential audio stream, into a category selected from a plurality of categories, the category corresponding to the first sound mode, a second sound mode identifier to identify a second sound mode for controlling the equalizer, from the plurality of sound modes, by segmenting a sound source into a music genre using a highlight extracted from stored music, and an equalizer controller to analyze the first sound mode and the second sound mode, select one of the first sound mode and the second sound mode, and control the equalizer according to the selected sound mode.