Benchmarking static code analyzers

Herter, Jörg, Kästner, Daniel, Christoph Mallon and Reinhard Wilhelm

Reliability Engineering and System Safety, 2019, vol. 188, issue C, 336-346

Abstract: We show that a widely used benchmark set for the comparison of static†=analysis tools exhibits an impressive number of weaknesses, and that the internationally accepted quantitative†=evaluation metrics may lead to useless results. The weaknesses in the benchmark set were identified by applying a sound static analysis to the programs in this set and carefully interpreting the results. We propose how to deal with weaknesses of the quantitative metrics and how to improve such benchmarks and the evaluation process, in particular for external evaluations, in which an ideally neutral institution does the evaluation, whose results potential clients can trust. We also show that sufficiently high quality of the test cases makes an automatic result evaluation possible.

Keywords: Static code analysis; Sound semantic analysis; C code; Safety-critical code; Benchmarking; Test case design; Abstract interpretation; Functional safety; Tool evaluation (search for similar items in EconPapers)
Date: 2019
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://www.sciencedirect.com/science/article/pii/S0951832018304721
Full text for ScienceDirect subscribers only

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:eee:reensy:v:188:y:2019:i:c:p:336-346

DOI: 10.1016/j.ress.2019.03.031

Access Statistics for this article

Reliability Engineering and System Safety is currently edited by Carlos Guedes Soares

More articles in Reliability Engineering and System Safety from Elsevier
Bibliographic data for series maintained by Catherine Liu ().