The invention discloses a machine learning anti-fraud monitoring system based on transaction data. The system comprises a management platform, an ETL module, a sampling engine, a stream processing engine, a training engine, a prediction engine and a decision engine. The stream processing engine rapidly extracts and calculates characteristics of the huge original transaction data through streamed big data processing, the representative characteristics are obtained from the huge original transaction data, and information in the data is sufficiently extracted. In the model training module, various machine learning models and ensemble learning frameworks optimized for the capital loss ratio and the black sample recall ratio are used, and a composite model optimized for an indicator is obtained. The over-fitting and unstable defects due to a single model are overcome, and the stability and the generalization ability of the model are improved; according to a preset update time, the model training module automatically obtain the latest data and trains the model again, accordingly the model keeps the effectiveness all the time, and the problem of model inefficiency due to fraud variation is avoided.