An architecture capable of carrying out variable length decoding for multiple video compression formats (e.g., MPEG1/2/4, H.263, H.264, Microsoft WMV9, and Sony Digital Video), is disclosed. In one embodiment, the VLD process is divided into two parts: flow control and table lookup. The flow control part can be performed by a low-cost microcontroller or other suitable processor, and the table lookup part is performed by hardware logic. With different firmware, the microcontroller handles flow control of all the existing video formats and can be adapted to accommodate new formats without any hardware change. Each piece of lookup table logic is connected to the microcontroller as extended instructions. In operation, during the decoding process, the flow control firmware executes one of these extended instructions whenever a table lookup operation is required. The architecture can be implemented, for example, as a system-on-chip decoder for use in HDTV applications and the like.