We have been fighting software vulnerabilities for years, but still cannot stop hackers and security researchers from discovering new vulnerabilities. In addition to software development cost limitations, the ability of automated software vulnerability mining technology has been unable to exceed the ability of manual code review is also the reason for this phenomenon. In order to understand the specific gap between these technologies and manual code review, I analyzed and tested some representative automatic software vulnerability mining tools. The representative tools I prepared include but are not limited to libFuzzer, AFL, KLEE, QSYM. I think these tools can fully reflect the capabilities and effectiveness of existing technologies. In research, I created the smallest example code where progarmmer can see the vulnerability at a glance, but these tools cannot find.

I have uploaded these example codes to GitHub https://github.com/arcslab/StutterFuzzers . These example codes prove the existence of the deficiencies and help you quickly start understanding, and I will share the deeper reasons.

At present, the state-of-the-art automatic software vulnerability mining technologies can be divided into four main types: unexpected behavior check, feedback-driven genetic algorithms, grammar or rule-based generate, and symbolic execute. Vulnerability mining tools involve at least one of these types or variants. In fact, all variants and combinations have fundamental deficiencies and make it impossible to go beyond manual code review capabilities. The most typical of these flaws is that the vulnerability is only in one of the hundreds of millions of executable code paths, and tools rarely traverse this path.

The typical feature of manual code review is to exponentially reduce the code path that needs to be reviewed in the process of continuously making and verifying assumptions. Although this method is not possible to cover all the code paths, it solves the problem of finding vulnerabilities from massive code paths. Inspired by manual code review, I proposed a technique called “variable constraint back propagation” based on LLVM bitcode and STP constraint solver. It uses the constraints generated by the assumption to filter the code paths that need to be traversed, and only the paths that can make the assumption true are traversed. This technology can find vulnerabilities that other technologies cannot find in the example codes, but to apply it to actual software, there are some practical difficulties to be solved.

Hackers and security researchers can use the results of my research to find untested code in software that has been sufficiently fuzzed by these tools, and invest their energy in areas where they are more likely to be productive. Software testers can also use my research results to improve the testing process in the software development life cycle. Some developers can even take advantage of these deficiencies to prevent hackers from finding vulnerabilities in software with automatic tools!