Software development and quality managers that are looking to measure the benefit of static analysis now have a platform to do just that. GrammaTech, under contract for the Department of Homeland Security (DHS) Science and Technology Directorate (S&T), has created independent real-world benchmarks that are now available in the Software Assurance Marketplace (SWAMP).
Several different synthetic benchmarks exist that can be used to measure how well static analysis tools perform in detecting bugs. However, many of these have limitations with the code paths typically being too simple. GrammaTech, under contract for DHS S&T, has created BugInjector, a tool that can inject Common Weakness Enumeration (CWE) based bug patterns into existing code bases, thus delivering real-world benchmarks. The BugInjector tool is available directly from GrammaTech to inject bugs into private code bases for training purposes. Additionally, four different real-world code bases (nginx, grep, sqlite, lighttpd) have been injected with bugs and are available through the Software Assurance Marketplace, enabling users to easily benchmark how well their static analysis tools are able to find these bugs in realistic code paths.
The Software Assurance Marketplace’s static analysis capabilities are available for use in the cloud or on-premise at no cost. Interested parties can sign up to use the SWAMP at mir-swamp.org and find the BugInjector test cases on the Resources page under Packages. After selecting a package and version containing a CWE of interest, users can run an assessment of the chosen “bug injected” software using one or more software assurance tools. GrammaTech CodeSonar is one of the commercial tools that is integrated into the SWAMP, along with many other open source static analysis tools. Users can also download BugInjector test cases to run against tools they are developing.