FUZZING TESTING
Fuzz testing or fuzzing is a software testing technique that provides invalid, unexpected or random data ("fuzz") to the inputs of a program. If the program fails (for example, by crashing or failing built-in code assertions), the defects can be noted.
File formats and network protocols are the most common targets of fuzz testing, but any type of program input can be fuzzed. Interesting inputs include environment variables, keyboard and mouse events, and sequences of API calls. Even items not normally considered "input" can be fuzzed, such as the contents of databases, shared memory, or the precise interleaving of threads.
For the purpose of security, input that crosses a trust boundary is often the most interesting[1]. For example, it is more important to fuzz code that handles a type of file commonly downloaded from the Web than it is to fuzz the parsing of a configuration file.
Fuzz testing is often used in large software development projects that employ black box testing. These projects usually have a budget to develop test tools, and fuzz testing is one of the techniques which offers a high benefit to cost ratio.
However, fuzz testing is not a substitute for exhaustive testing or formal methods: it can only provide a random sample of the system's behavior, and in many cases passing a fuzz test may only demonstrate that a piece of software handles exceptions without crashing, rather than behaving correctly. Thus, fuzz testing can only be regarded as a bug-finding tool rather than an assurance of quality.
As a gross measurement of reliability, fuzzing can suggest which parts of a program should get special attention, in the form of a code audit, application of static analysis, or partial rewrites.
Techniques
The oldest and simplest form of fuzzing, sending a stream of random bits to software, continues to find bugs in command-line applications[2]. Another common technique that is easy to implement is mutating existing input (e.g. files from a test suite) by flipping bits at random or moving blocks of the file around.
The most successful fuzzers have detailed understanding of the format or protocol being tested. The understanding can be based on aspecification, or it can be created heuristically from examples using a tool such as Sequitur[3]. These fuzzers can generate test casesfrom scratch, or they can mutate examples from test suites or real life. They can concentrate on valid or invalid input, with mostly-validinput tending to trigger the "deepest" error cases.
Fuzz testing can be combined with other testing techniques. White-box fuzzing uses symbolic execution and constraint solving[4].Evolutionary fuzzing leverages feedback from code coverage[5], effectively automating the approach of exploratory testing.
Straight-up failures such as crashes, assertion failures, and memory leaks are easy to detect. The use of a memory debugger can help find bugs too subtle to always crash.
Fuzz testing is especially useful against large C or C++ applications, where any bug affecting memory safety is likely to be a severevulnerability. It is these security concerns that motivate the development of most fuzzers.
Since fuzzing often generates invalid input, it is especially good at testing error-handling routines, which are important for software that does not control its input. As such, simple fuzzing can be thought of as a way to automate negative testing. More sophisticated fuzzing tests more "main-line" code, along with error paths deep within it.
Fuzzing can also find some types of "correctness" bugs. For example, it can be used to find incorrect-serialization bugs by complaining whenever a program's serializer emits something that the same program's parser rejects[6]. It can also find unintentional differences between two versions of a program[7] or between two implementations of the same specification[8].
As a practical matter, developers need to reproduce errors in order to fix them. For this reason, almost all fuzz testing makes a record of the data it manufactures, usually before applying it to the software, so that if the computer fails dramatically, the test data is preserved. If the fuzz stream is pseudo-random number generated it may be easier to store the seed value to reproduce the fuzz attempt.
Once a bug found through fuzzing is reproduced, it is often desirable to produce a simple test case to make the issue easier to understand anddebug. A simple testcase may also be faster and therefore more suitable for inclusion in a test suite that is run frequently. It can even help to hide the way in which the bug was found. Some fuzzers are designed to work well with testcase reduction programs such as Delta orLithium.
The main problem with fuzzing to find program faults is that it generally only finds very simple faults. The problem itself is exponential and every fuzzer takes shortcuts to find something interesting in a timeframe that a human cares about. A primitive fuzzer may have poor code coverage; for example, if the input includes a checksum which is not properly updated to match other random changes, only the checksum validation code will be verified. Code coverage tools are often used to estimate how "well" a fuzzer works, but these are only guidelines to fuzzer quality. Every fuzzer can be expected to find a different set of bugs.
On the other hand, bugs found using fuzz testing are sometimes severe, exploitable bugs that could be used by a real attacker. This has become more common as fuzz testing has become more widely known, as the same techniques and tools are now used by attackers to exploit deployed software. This is a major advantage over binary or source auditing, or even fuzzing's close cousin, fault injection, which often relies on artificial fault conditions that are difficult or impossible to exploit.
The randomness of inputs used in fuzzing is often seen as a disadvantage, as catching a boundary value condition with random inputs is highly unlikely.
Fuzz testing is enhances software security and software safety because it often finds odd oversights and defects which human testers would fail to find, and even careful human test designers would fail to create tests for.