Ensemble one-class detection for PCIe bus security: a comparative study of algorithm

Abstract

This paper presents a systematic comparative study of four one-class detection algorithms — a Mahalanobis detector, Isolation Forest, Deep Autoencoder, and their ensemble — for DMA anomaly detection in PCIe device traffic. All algorithms were trained and evaluated on a unified TLP traffic dataset (7,300 time windows, 5 attack classes) collected in a QEMU 8.1 environment. The ORensemble detector (IF + Mahalanobis) achieved the best results: Recall = 0.905, FPR = 0.007, F₁ = 0.943, AUC-ROC = 0.961, without requiring any labeled attack samples.