The invention discloses a universal floating point matrix processor hardware structure based on an FPGA (field programmable gate array). The universal floating point matrix processor hardware structure comprises a plurality of matrix operation accelerators, a plurality of shared matrix buffer memories, a main processor, an instruction dispatcher, a direct access controller, an arbiter, an external memory and a memory management unit, the shared matrix buffer memories provide universal memory spaces and support high-speed communication on chips between the accelerators, the main processor transmits instructions, dispatches operation and calculates addresses, the instruction dispatcher inspects whether the instructions have occupying conflict or not, the direct access controller finishes data transmission, the arbiter distributes data, the external memory provides buffer, and the memory management unit provides an access interface. The hardware structure can realize 'matrix-matrix' operation on a universal platform, software programs can be modified on the main processor according to different applications, a matrix operation accelerator structure can be conveniently inserted to support various matrix operations, the universal floating point matrix processor hardware structure has high design flexibility and has the advantages of lower energy consumption and higher calculated performance as compared with an embedded processor performing the matrix operations in the past.