A new research paper, "Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners," introduces a framework to evaluate the effectiveness of security scanners for machine learning models. The paper, published on arXiv, highlights limitations in traditional evaluation metrics like F1 scores, which often only consider cases where a scanner produces a usable security judgment. The authors argue that a more complete assessment requires distinguishing between a scanner's ability to complete an analysis and its accuracy once a judgment is made.
The researchers evaluated three open-source security scanners: ModelScan, ModelAudit, and Fickling. Their benchmark used a synthetic corpus of 170 artifacts, primarily focusing on Pickle and PyTorch formats. This corpus included 145 specimen families, with 135 having a known binary security ground truth, and 10 intentionally malformed artifacts without labels.
The evaluation explicitly measured several distinct outcomes: non-N/A coverage, analysis completion, definitive security decisions, non-security findings, and unsupported outcomes. This detailed approach aims to provide a clearer picture of scanner performance than metrics that solely focus on the accuracy of definitive judgments.
On the 135 labeled families, ModelAudit provided definitive security decisions for all 135 families, achieving 100% coverage. Fickling made definitive decisions for 110 families, or 81.5% of the corpus. ModelScan provided definitive security decisions for 67 families, covering 49.6%.
While ModelScan achieved perfect precision, recall, and F1 scores when it did make a definitive judgment, its lower coverage meant it remained silent on a significant portion of the test cases. Notably, for 48 malicious families where ModelScan failed to complete its analysis, both ModelAudit and Fickling successfully detected issues consistent with the ground truth. This finding suggests that a scanner's ability to complete an analysis is as important as its accuracy when it does produce a result.
The paper emphasizes that conventional F1 scores can be misleading if they do not account for instances where a scanner fails to render any judgment. A tool that appears to have high accuracy on the cases it processes may still leave critical security gaps if it frequently fails to analyze certain inputs. This distinction between "judgment accuracy" and "judgment availability" is a central contribution of the new framework.
ModelAudit, developed by Promptfoo, is an open-source scanner that supports over 42 formats and includes built-in CVE detection rules. It focuses on flagging unsafe loading behaviors, known vulnerabilities, and suspicious artifacts without executing the model. Previous comparisons, such as one conducted in July 2025, also highlighted ModelAudit's broader format coverage compared to ModelScan. That comparison showed ModelAudit detecting 16 issues across 11 test files, while ModelScan detected 3.
Fickling, maintained by Trail of Bits, is a pickle file scanner and decompiler. It can decompile pickle streams into readable Python and incorporates an allowlist-based scanner. The research indicated that Fickling did not identify any unique true-positive families beyond those already found by ModelAudit and ModelScan combined.
The researchers conclude that separating judgment accuracy from judgment availability is essential for a complete evaluation of AI model security scanners. They also highlight the importance of distinguishing incremental detection coverage from tool-level redundancy. This research aims to provide a more nuanced understanding of scanner performance, moving beyond metrics that might obscure significant blind spots in security coverage.