How fuzz testing works

By:

Fuzz testing is a simple technique that can have a profound effect on your code quality. In this article, Elliotte Rusty Harold shows what happens when he deliberately injects random bad data into an application to see what breaks. He also explains how to use defensive coding techniques such as checksums, XML data storage, and code verification to harden your programs against random data. He concludes with an exercise in thinking like a code cracker -- a crucial technique for defending your code

For years, I've been astounded by the number of corrupt files that can crash Microsoft Word. A few bytes out of place and the whole application goes down in flames. On older, non-memory-protected operating systems, the whole computer usually went down with it. Why can't Word recognize when it's received bad data and simply put up an error message? Why does it corrupt its own stack and heap just because a few bits got twiddled? Of course, Word is hardly the only program that behaves atrociously in the face of malformed files.

This article introduces you to a technique that attempts to avert just this sort of disaster. In fuzz testing, you attack a program with random bad data (aka fuzz), then wait to see what breaks. The trick of fuzz testing is that it isn't logical: Rather than attempting to guess what data is likely to provoke a crash (as a human tester might do), an automated fuzz test simply throws as much random gibberish at a program as possible. The failure modes identified by such testing usually come as a complete shock to programmers because no logical person would ever conceive of them.

Fuzz testing is a simple technique, but it can nonetheless reveal important bugs in your programs. It can identify real-world failure modes and signal potential avenues of attack that should be plugged before your software ships.

How fuzz testing works
Fuzz testing is a very simple procedure to implement:
Prepare a correct file to input to your program.
Replace some part of the file with random data.
Open the file with the program.
See what breaks.


You can vary the random data in any number of ways. For example, you might randomize the entire file rather than replacing just a part of it. You could limit the file to ASCII text or non-zero bytes. Any way you slice it, the key is to throw a lot of random data at an application and see what fails.

While you can do initial tests manually, you should really automate fuzzing for maximum effect. In this case, you first need to define the proper error behavior for the application when faced with corrupt input. (If you discover the program hasn't bothered to define what happens when the input data is corrupt, well, there's your first bug.) Then you simply pass random data into the program until you find a file that doesn't trigger the proper error dialog, message, exception, etc. Store and log that file so you can reproduce the problem later. Repeat.

Although fuzz testing usually requires some manual coding, there are tools that can help. For example, Listing 1 shows a simple Java™ class that randomly modifies a certain length of a file. I usually like to start fuzzing somewhere after the first few bytes because programs seem more likely to notice an early mistake than a later one. (You want to find the errors the program doesn't check, not the ones it does.)

Listing 1. A class that replaces part of a file with random data

import java.io.*;
import java.security.SecureRandom;
import java.util.Random;

public class Fuzzer {

private Random random = new SecureRandom();
private int count = 1;

public File fuzz(File in, int start, int length) throws IOException
{
byte[] data = new byte[(int) in.length()];
DataInputStream din = new DataInputStream(new FileInputStream(in));
din.readFully(data);
fuzz(data, start, length);
String name = "fuzz_" + count + "_" + in.getName();
File fout = new File(name);
FileOutputStream out = new FileOutputStream(fout);
out.write(data);
out.close();
din.close();
count++;
return fout;
}

// Modifies byte array in place
public void fuzz(byte[] in, int start, int length) {

byte[] fuzz = new byte[length];
random.nextBytes(fuzz);
System.arraycopy(fuzz, 0, in, start, fuzz.length);

}

}

Full article...

Other Resource

... to read more articles, visit http://sqa.fyicenter.com/art/