LIBIHT: A Hardware-Based Approach to Efficient and Evasion-Resistant Dynamic Binary Analysis

Abstract

Dynamic program analysis is invaluable for malware detection, debugging, and performance profiling. However, software-based instrumentation incurs high overhead and can be evaded by anti-analysis techniques. In this paper, we propose LIBIHT, a hardware-assisted tracing framework that leverages on-CPU branch tracing features (Intel Last Branch Record and Branch Trace Store) to efficiently capture program control-flow with minimal performance impact. Our approach reconstructs control-flow graphs (CFGs) by collecting hardware generated branch execution data in the kernel, preserving program behavior against evasive malware. We implement LIBIHT as an OS kernel module and user-space library, and evaluate it on both benign benchmark programs and adversarial anti- instrumentation samples. Our results indicate that LIBIHT reduces runtime overhead by over 150× compared to Intel Pin (7× vs 1,053× slowdowns), while achieving high fidelity in CFG reconstruction (capturing over 99% of execution basic blocks and edges). Although this hardware-assisted approach sacrifices the richer semantic detail available from full software instrumentation by capturing only branch addresses, this trade- off is acceptable for many applications where performance and low detectability are paramount. Our findings show that hardware-based tracing substantially improves performance while reducing detection risk and enabling dynamic analysis with minimal interference.