diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/FAQ.md | 5 | ||||
| -rw-r--r-- | docs/afl-fuzz_approach.md | 542 | ||||
| -rw-r--r-- | docs/best_practices.md | 24 | ||||
| -rw-r--r-- | docs/beyond_crashes.md | 23 | ||||
| -rw-r--r-- | docs/binaryonly_fuzzing.md | 225 | ||||
| -rw-r--r-- | docs/branches.md | 11 | ||||
| -rw-r--r-- | docs/choosing_testcases.md | 19 | ||||
| -rw-r--r-- | docs/ci_fuzzing.md | 29 | ||||
| -rw-r--r-- | docs/common_sense_risks.md | 36 | ||||
| -rw-r--r-- | docs/custom_mutators.md | 6 | ||||
| -rw-r--r-- | docs/env_variables.md | 27 | ||||
| -rw-r--r-- | docs/features.md | 96 | ||||
| -rw-r--r-- | docs/fuzzing_binary-only_targets.md | 300 | ||||
| -rw-r--r-- | docs/fuzzing_expert.md | 626 | ||||
| -rw-r--r-- | docs/fuzzing_in_depth.md | 853 | ||||
| -rw-r--r-- | docs/important_changes.md | 4 | ||||
| -rw-r--r-- | docs/interpreting_output.md | 71 | ||||
| -rw-r--r-- | docs/known_limitations.md | 36 | ||||
| -rw-r--r-- | docs/parallel_fuzzing.md | 258 | ||||
| -rw-r--r-- | docs/perf_tips.md | 209 | ||||
| -rw-r--r-- | docs/sister_projects.md | 319 | ||||
| -rw-r--r-- | docs/status_screen.md | 444 | ||||
| -rw-r--r-- | docs/technical_details.md | 550 | ||||
| -rw-r--r-- | docs/third_party_tools.md | 57 | ||||
| -rw-r--r-- | docs/tools.md | 33 | ||||
| -rw-r--r-- | docs/triaging_crashes.md | 46 | ||||
| -rw-r--r-- | docs/tutorials.md | 14 |
27 files changed, 1788 insertions, 3075 deletions
diff --git a/docs/FAQ.md b/docs/FAQ.md index 68ca3bad..34ed4cf5 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -83,7 +83,8 @@ If you find an interesting or important question missing, submit it via However, if there is only the binary program and no source code available, then the standard non-instrumented mode is not effective. - To learn how these binaries can be fuzzed, read [binaryonly_fuzzing.md](binaryonly_fuzzing.md). + To learn how these binaries can be fuzzed, read + [fuzzing_binary-only_targets.md](fuzzing_binary-only_targets.md). </p></details> <details> @@ -143,7 +144,7 @@ If you find an interesting or important question missing, submit it via Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /prg/tmp/llvm-project/build/bin - clang-13: note: diagnostic msg: + clang-13: note: diagnostic msg: ******************** ``` diff --git a/docs/afl-fuzz_approach.md b/docs/afl-fuzz_approach.md index 5652816b..e0d5a1c9 100644 --- a/docs/afl-fuzz_approach.md +++ b/docs/afl-fuzz_approach.md @@ -1,37 +1,541 @@ # The afl-fuzz approach -American Fuzzy Lop is a brute-force fuzzer coupled with an exceedingly simple -but rock-solid instrumentation-guided genetic algorithm. It uses a modified -form of edge coverage to effortlessly pick up subtle, local-scale changes to -program control flow. +AFL++ is a brute-force fuzzer coupled with an exceedingly simple but rock-solid +instrumentation-guided genetic algorithm. It uses a modified form of edge +coverage to effortlessly pick up subtle, local-scale changes to program control +flow. Simplifying a bit, the overall algorithm can be summed up as: - 1) Load user-supplied initial test cases into the queue, +1) Load user-supplied initial test cases into the queue. - 2) Take the next input file from the queue, +2) Take the next input file from the queue. - 3) Attempt to trim the test case to the smallest size that doesn't alter - the measured behavior of the program, +3) Attempt to trim the test case to the smallest size that doesn't alter the + measured behavior of the program. - 4) Repeatedly mutate the file using a balanced and well-researched variety - of traditional fuzzing strategies, +4) Repeatedly mutate the file using a balanced and well-researched variety of + traditional fuzzing strategies. - 5) If any of the generated mutations resulted in a new state transition - recorded by the instrumentation, add mutated output as a new entry in the - queue. +5) If any of the generated mutations resulted in a new state transition recorded + by the instrumentation, add mutated output as a new entry in the queue. - 6) Go to 2. +6) Go to 2. The discovered test cases are also periodically culled to eliminate ones that have been obsoleted by newer, higher-coverage finds; and undergo several other instrumentation-driven effort minimization steps. As a side result of the fuzzing process, the tool creates a small, -self-contained corpus of interesting test cases. These are extremely useful -for seeding other, labor- or resource-intensive testing regimes - for example, -for stress-testing browsers, office applications, graphics suites, or -closed-source tools. +self-contained corpus of interesting test cases. These are extremely useful for +seeding other, labor- or resource-intensive testing regimes - for example, for +stress-testing browsers, office applications, graphics suites, or closed-source +tools. The fuzzer is thoroughly tested to deliver out-of-the-box performance far -superior to blind fuzzing or coverage-only tools. \ No newline at end of file +superior to blind fuzzing or coverage-only tools. + +## Understanding the status screen + +This chapter provides an overview of the status screen - plus tips for +troubleshooting any warnings and red text shown in the UI. + +For the general instruction manual, see [README.md](../README.md). + +### A note about colors + +The status screen and error messages use colors to keep things readable and +attract your attention to the most important details. For example, red almost +always means "consult this doc" :-) + +Unfortunately, the UI will only render correctly if your terminal is using +traditional un*x palette (white text on black background) or something close to +that. + +If you are using inverse video, you may want to change your settings, say: + +- For GNOME Terminal, go to `Edit > Profile` preferences, select the "colors" + tab, and from the list of built-in schemes, choose "white on black". +- For the MacOS X Terminal app, open a new window using the "Pro" scheme via the + `Shell > New Window` menu (or make "Pro" your default). + +Alternatively, if you really like your current colors, you can edit config.h to +comment out USE_COLORS, then do `make clean all`. + +We are not aware of any other simple way to make this work without causing other +side effects - sorry about that. + +With that out of the way, let's talk about what's actually on the screen... + +### The status bar + +``` +american fuzzy lop ++3.01a (default) [fast] {0} +``` + +The top line shows you which mode afl-fuzz is running in (normal: "american +fuzzy lop", crash exploration mode: "peruvian rabbit mode") and the version of +AFL++. Next to the version is the banner, which, if not set with -T by hand, +will either show the binary name being fuzzed, or the -M/-S main/secondary name +for parallel fuzzing. Second to last is the power schedule mode being run +(default: fast). Finally, the last item is the CPU id. + +### Process timing + +``` + +----------------------------------------------------+ + | run time : 0 days, 8 hrs, 32 min, 43 sec | + | last new path : 0 days, 0 hrs, 6 min, 40 sec | + | last uniq crash : none seen yet | + | last uniq hang : 0 days, 1 hrs, 24 min, 32 sec | + +----------------------------------------------------+ +``` + +This section is fairly self-explanatory: it tells you how long the fuzzer has +been running and how much time has elapsed since its most recent finds. This is +broken down into "paths" (a shorthand for test cases that trigger new execution +patterns), crashes, and hangs. + +When it comes to timing: there is no hard rule, but most fuzzing jobs should be +expected to run for days or weeks; in fact, for a moderately complex project, +the first pass will probably take a day or so. Every now and then, some jobs +will be allowed to run for months. + +There's one important thing to watch out for: if the tool is not finding new +paths within several minutes of starting, you're probably not invoking the +target binary correctly and it never gets to parse the input files we're +throwing at it; other possible explanations are that the default memory limit +(`-m`) is too restrictive and the program exits after failing to allocate a +buffer very early on; or that the input files are patently invalid and always +fail a basic header check. + +If there are no new paths showing up for a while, you will eventually see a big +red warning in this section, too :-) + +### Overall results + +``` + +-----------------------+ + | cycles done : 0 | + | total paths : 2095 | + | uniq crashes : 0 | + | uniq hangs : 19 | + +-----------------------+ +``` + +The first field in this section gives you the count of queue passes done so far +- that is, the number of times the fuzzer went over all the interesting test + cases discovered so far, fuzzed them, and looped back to the very beginning. + Every fuzzing session should be allowed to complete at least one cycle; and + ideally, should run much longer than that. + +As noted earlier, the first pass can take a day or longer, so sit back and +relax. + +To help make the call on when to hit `Ctrl-C`, the cycle counter is color-coded. +It is shown in magenta during the first pass, progresses to yellow if new finds +are still being made in subsequent rounds, then blue when that ends - and +finally, turns green after the fuzzer hasn't been seeing any action for a longer +while. + +The remaining fields in this part of the screen should be pretty obvious: +there's the number of test cases ("paths") discovered so far, and the number of +unique faults. The test cases, crashes, and hangs can be explored in real-time +by browsing the output directory, see +[#interpreting-output](#interpreting-output). + +### Cycle progress + +``` + +-------------------------------------+ + | now processing : 1296 (61.86%) | + | paths timed out : 0 (0.00%) | + +-------------------------------------+ +``` + +This box tells you how far along the fuzzer is with the current queue cycle: it +shows the ID of the test case it is currently working on, plus the number of +inputs it decided to ditch because they were persistently timing out. + +The "*" suffix sometimes shown in the first line means that the currently +processed path is not "favored" (a property discussed later on). + +### Map coverage + +``` + +--------------------------------------+ + | map density : 10.15% / 29.07% | + | count coverage : 4.03 bits/tuple | + +--------------------------------------+ +``` + +The section provides some trivia about the coverage observed by the +instrumentation embedded in the target binary. + +The first line in the box tells you how many branch tuples we have already hit, +in proportion to how much the bitmap can hold. The number on the left describes +the current input; the one on the right is the value for the entire input +corpus. + +Be wary of extremes: + +- Absolute numbers below 200 or so suggest one of three things: that the program + is extremely simple; that it is not instrumented properly (e.g., due to being + linked against a non-instrumented copy of the target library); or that it is + bailing out prematurely on your input test cases. The fuzzer will try to mark + this in pink, just to make you aware. +- Percentages over 70% may very rarely happen with very complex programs that + make heavy use of template-generated code. Because high bitmap density makes + it harder for the fuzzer to reliably discern new program states, we recommend + recompiling the binary with `AFL_INST_RATIO=10` or so and trying again (see + [env_variables.md](env_variables.md)). The fuzzer will flag high percentages + in red. Chances are, you will never see that unless you're fuzzing extremely + hairy software (say, v8, perl, ffmpeg). + +The other line deals with the variability in tuple hit counts seen in the +binary. In essence, if every taken branch is always taken a fixed number of +times for all the inputs we have tried, this will read `1.00`. As we manage to +trigger other hit counts for every branch, the needle will start to move toward +`8.00` (every bit in the 8-bit map hit), but will probably never reach that +extreme. + +Together, the values can be useful for comparing the coverage of several +different fuzzing jobs that rely on the same instrumented binary. + +### Stage progress + +``` + +-------------------------------------+ + | now trying : interest 32/8 | + | stage execs : 3996/34.4k (11.62%) | + | total execs : 27.4M | + | exec speed : 891.7/sec | + +-------------------------------------+ +``` + +This part gives you an in-depth peek at what the fuzzer is actually doing right +now. It tells you about the current stage, which can be any of: + +- calibration - a pre-fuzzing stage where the execution path is examined to + detect anomalies, establish baseline execution speed, and so on. Executed very + briefly whenever a new find is being made. +- trim L/S - another pre-fuzzing stage where the test case is trimmed to the + shortest form that still produces the same execution path. The length (L) and + stepover (S) are chosen in general relationship to file size. +- bitflip L/S - deterministic bit flips. There are L bits toggled at any given + time, walking the input file with S-bit increments. The current L/S variants + are: `1/1`, `2/1`, `4/1`, `8/8`, `16/8`, `32/8`. +- arith L/8 - deterministic arithmetics. The fuzzer tries to subtract or add + small integers to 8-, 16-, and 32-bit values. The stepover is always 8 bits. +- interest L/8 - deterministic value overwrite. The fuzzer has a list of known + "interesting" 8-, 16-, and 32-bit values to try. The stepover is 8 bits. +- extras - deterministic injection of dictionary terms. This can be shown as + "user" or "auto", depending on whether the fuzzer is using a user-supplied + dictionary (`-x`) or an auto-created one. You will also see "over" or + "insert", depending on whether the dictionary words overwrite existing data or + are inserted by offsetting the remaining data to accommodate their length. +- havoc - a sort-of-fixed-length cycle with stacked random tweaks. The + operations attempted during this stage include bit flips, overwrites with + random and "interesting" integers, block deletion, block duplication, plus + assorted dictionary-related operations (if a dictionary is supplied in the + first place). +- splice - a last-resort strategy that kicks in after the first full queue cycle + with no new paths. It is equivalent to 'havoc', except that it first splices + together two random inputs from the queue at some arbitrarily selected + midpoint. +- sync - a stage used only when `-M` or `-S` is set (see + [parallel_fuzzing.md](parallel_fuzzing.md)). No real fuzzing is involved, but + the tool scans the output from other fuzzers and imports test cases as + necessary. The first time this is done, it may take several minutes or so. + +The remaining fields should be fairly self-evident: there's the exec count +progress indicator for the current stage, a global exec counter, and a benchmark +for the current program execution speed. This may fluctuate from one test case +to another, but the benchmark should be ideally over 500 execs/sec most of the +time - and if it stays below 100, the job will probably take very long. + +The fuzzer will explicitly warn you about slow targets, too. If this happens, +see the [perf_tips.md](perf_tips.md) file included with the fuzzer for ideas on +how to speed things up. + +### Findings in depth + +``` + +--------------------------------------+ + | favored paths : 879 (41.96%) | + | new edges on : 423 (20.19%) | + | total crashes : 0 (0 unique) | + | total tmouts : 24 (19 unique) | + +--------------------------------------+ +``` + +This gives you several metrics that are of interest mostly to complete nerds. +The section includes the number of paths that the fuzzer likes the most based on +a minimization algorithm baked into the code (these will get considerably more +air time), and the number of test cases that actually resulted in better edge +coverage (versus just pushing the branch hit counters up). There are also +additional, more detailed counters for crashes and timeouts. + +Note that the timeout counter is somewhat different from the hang counter; this +one includes all test cases that exceeded the timeout, even if they did not +exceed it by a margin sufficient to be classified as hangs. + +### Fuzzing strategy yields + +``` + +-----------------------------------------------------+ + | bit flips : 57/289k, 18/289k, 18/288k | + | byte flips : 0/36.2k, 4/35.7k, 7/34.6k | + | arithmetics : 53/2.54M, 0/537k, 0/55.2k | + | known ints : 8/322k, 12/1.32M, 10/1.70M | + | dictionary : 9/52k, 1/53k, 1/24k | + |havoc/splice : 1903/20.0M, 0/0 | + |py/custom/rq : unused, 53/2.54M, unused | + | trim/eff : 20.31%/9201, 17.05% | + +-----------------------------------------------------+ +``` + +This is just another nerd-targeted section keeping track of how many paths we +have netted, in proportion to the number of execs attempted, for each of the +fuzzing strategies discussed earlier on. This serves to convincingly validate +assumptions about the usefulness of the various approaches taken by afl-fuzz. + +The trim strategy stats in this section are a bit different than the rest. The +first number in this line shows the ratio of bytes removed from the input files; +the second one corresponds to the number of execs needed to achieve this goal. +Finally, the third number shows the proportion of bytes that, although not +possible to remove, were deemed to have no effect and were excluded from some of +the more expensive deterministic fuzzing steps. + +Note that when deterministic mutation mode is off (which is the default because +it is not very efficient) the first five lines display "disabled (default, +enable with -D)". + +Only what is activated will have counter shown. + +### Path geometry + +``` + +---------------------+ + | levels : 5 | + | pending : 1570 | + | pend fav : 583 | + | own finds : 0 | + | imported : 0 | + | stability : 100.00% | + +---------------------+ +``` + +The first field in this section tracks the path depth reached through the guided +fuzzing process. In essence: the initial test cases supplied by the user are +considered "level 1". The test cases that can be derived from that through +traditional fuzzing are considered "level 2"; the ones derived by using these as +inputs to subsequent fuzzing rounds are "level 3"; and so forth. The maximum +depth is therefore a rough proxy for how much value you're getting out of the +instrumentation-guided approach taken by afl-fuzz. + +The next field shows you the number of inputs that have not gone through any +fuzzing yet. The same stat is also given for "favored" entries that the fuzzer +really wants to get to in this queue cycle (the non-favored entries may have to +wait a couple of cycles to get their chance). + +Next, we have the number of new paths found during this fuzzing section and +imported from other fuzzer instances when doing parallelized fuzzing; and the +extent to which identical inputs appear to sometimes produce variable behavior +in the tested binary. + +That last bit is actually fairly interesting: it measures the consistency of +observed traces. If a program always behaves the same for the same input data, +it will earn a score of 100%. When the value is lower but still shown in purple, +the fuzzing process is unlikely to be negatively affected. If it goes into red, +you may be in trouble, since AFL will have difficulty discerning between +meaningful and "phantom" effects of tweaking the input file. + +Now, most targets will just get a 100% score, but when you see lower figures, +there are several things to look at: + +- The use of uninitialized memory in conjunction with some intrinsic sources of + entropy in the tested binary. Harmless to AFL, but could be indicative of a + security bug. +- Attempts to manipulate persistent resources, such as left over temporary files + or shared memory objects. This is usually harmless, but you may want to + double-check to make sure the program isn't bailing out prematurely. Running + out of disk space, SHM handles, or other global resources can trigger this, + too. +- Hitting some functionality that is actually designed to behave randomly. + Generally harmless. For example, when fuzzing sqlite, an input like `select + random();` will trigger a variable execution path. +- Multiple threads executing at once in semi-random order. This is harmless when + the 'stability' metric stays over 90% or so, but can become an issue if not. + Here's what to try: + * Use afl-clang-fast from [instrumentation](../instrumentation/) - it uses a + thread-local tracking model that is less prone to concurrency issues, + * See if the target can be compiled or run without threads. Common + `./configure` options include `--without-threads`, `--disable-pthreads`, or + `--disable-openmp`. + * Replace pthreads with GNU Pth (https://www.gnu.org/software/pth/), which + allows you to use a deterministic scheduler. +- In persistent mode, minor drops in the "stability" metric can be normal, + because not all the code behaves identically when re-entered; but major dips + may signify that the code within `__AFL_LOOP()` is not behaving correctly on + subsequent iterations (e.g., due to incomplete clean-up or reinitialization of + the state) and that most of the fuzzing effort goes to waste. + +The paths where variable behavior is detected are marked with a matching entry +in the `<out_dir>/queue/.state/variable_behavior/` directory, so you can look +them up easily. + +### CPU load + +``` + [cpu: 25%] +``` + +This tiny widget shows the apparent CPU utilization on the local system. It is +calculated by taking the number of processes in the "runnable" state, and then +comparing it to the number of logical cores on the system. + +If the value is shown in green, you are using fewer CPU cores than available on +your system and can probably parallelize to improve performance; for tips on how +to do that, see [parallel_fuzzing.md](parallel_fuzzing.md). + +If the value is shown in red, your CPU is *possibly* oversubscribed, and running +additional fuzzers may not give you any benefits. + +Of course, this benchmark is very simplistic; it tells you how many processes +are ready to run, but not how resource-hungry they may be. It also doesn't +distinguish between physical cores, logical cores, and virtualized CPUs; the +performance characteristics of each of these will differ quite a bit. + +If you want a more accurate measurement, you can run the `afl-gotcpu` utility +from the command line. + +## Interpreting output + +See [#understanding-the-status-screen](#understanding-the-status-screen) for +information on how to interpret the displayed stats and monitor the health of +the process. Be sure to consult this file especially if any UI elements are +highlighted in red. + +The fuzzing process will continue until you press Ctrl-C. At a minimum, you want +to allow the fuzzer to complete one queue cycle, which may take anywhere from a +couple of hours to a week or so. + +There are three subdirectories created within the output directory and updated +in real-time: + +- queue/ - test cases for every distinctive execution path, plus all the + starting files given by the user. This is the synthesized corpus + mentioned in section 2. + + Before using this corpus for any other purposes, you can shrink + it to a smaller size using the afl-cmin tool. The tool will find + a smaller subset of files offering equivalent edge coverage. + +- crashes/ - unique test cases that cause the tested program to receive a fatal + signal (e.g., SIGSEGV, SIGILL, SIGABRT). The entries are grouped by + the received signal. + +- hangs/ - unique test cases that cause the tested program to time out. The + default time limit before something is classified as a hang is the + larger of 1 second and the value of the -t parameter. The value can + be fine-tuned by setting AFL_HANG_TMOUT, but this is rarely + necessary. + +Crashes and hangs are considered "unique" if the associated execution paths +involve any state transitions not seen in previously-recorded faults. If a +single bug can be reached in multiple ways, there will be some count inflation +early in the process, but this should quickly taper off. + +The file names for crashes and hangs are correlated with the parent, non-faulting +queue entries. This should help with debugging. + +## Visualizing + +If you have gnuplot installed, you can also generate some pretty graphs for any +active fuzzing task using afl-plot. For an example of how this looks like, see +[https://lcamtuf.coredump.cx/afl/plot/](https://lcamtuf.coredump.cx/afl/plot/). + +You can also manually build and install afl-plot-ui, which is a helper utility +for showing the graphs generated by afl-plot in a graphical window using GTK. +You can build and install it as follows: + +```shell +sudo apt install libgtk-3-0 libgtk-3-dev pkg-config +cd utils/plot_ui +make +cd ../../ +sudo make install +``` + + +### Addendum: status and plot files + +For unattended operation, some of the key status screen information can be also +found in a machine-readable format in the fuzzer_stats file in the output +directory. This includes: + +- `start_time` - unix time indicating the start time of afl-fuzz +- `last_update` - unix time corresponding to the last update of this file +- `run_time` - run time in seconds to the last update of this file +- `fuzzer_pid` - PID of the fuzzer process +- `cycles_done` - queue cycles completed so far +- `cycles_wo_finds` - number of cycles without any new paths found +- `execs_done` - number of execve() calls attempted +- `execs_per_sec` - overall number of execs per second +- `paths_total` - total number of entries in the queue +- `paths_favored` - number of queue entries that are favored +- `paths_found` - number of entries discovered through local fuzzing +- `paths_imported` - number of entries imported from other instances +- `max_depth` - number of levels in the generated data set +- `cur_path` - currently processed entry number +- `pending_favs` - number of favored entries still waiting to be fuzzed +- `pending_total` - number of all entries waiting to be fuzzed +- `variable_paths` - number of test cases showing variable behavior +- `stability` - percentage of bitmap bytes that behave consistently +- `bitmap_cvg` - percentage of edge coverage found in the map so far +- `unique_crashes` - number of unique crashes recorded +- `unique_hangs` - number of unique hangs encountered +- `last_path` - seconds since the last path was found +- `last_crash` - seconds since the last crash was found +- `last_hang` - seconds since the last hang was found +- `execs_since_crash` - execs since the last crash was found +- `exec_timeout` - the -t command line value +- `slowest_exec_ms` - real time of the slowest execution in ms +- `peak_rss_mb` - max rss usage reached during fuzzing in MB +- `edges_found` - how many edges have been found +- `var_byte_count` - how many edges are non-deterministic +- `afl_banner` - banner text (e.g. the target name) +- `afl_version` - the version of AFL used +- `target_mode` - default, persistent, qemu, unicorn, non-instrumented +- `command_line` - full command line used for the fuzzing session + +Most of these map directly to the UI elements discussed earlier on. + +On top of that, you can also find an entry called `plot_data`, containing a +plottable history for most of these fields. If you have gnuplot installed, you +can turn this into a nice progress report with the included `afl-plot` tool. + +### Addendum: automatically sending metrics with StatsD + +In a CI environment or when running multiple fuzzers, it can be tedious to log +into each of them or deploy scripts to read the fuzzer statistics. Using +`AFL_STATSD` (and the other related environment variables `AFL_STATSD_HOST`, +`AFL_STATSD_PORT`, `AFL_STATSD_TAGS_FLAVOR`) you can automatically send metrics +to your favorite StatsD server. Depending on your StatsD server, you will be +able to monitor, trigger alerts, or perform actions based on these metrics (e.g: +alert on slow exec/s for a new build, threshold of crashes, time since last +crash > X, etc). + +The selected metrics are a subset of all the metrics found in the status and in +the plot file. The list is the following: `cycle_done`, `cycles_wo_finds`, +`execs_done`,`execs_per_sec`, `paths_total`, `paths_favored`, `paths_found`, +`paths_imported`, `max_depth`, `cur_path`, `pending_favs`, `pending_total`, +`variable_paths`, `unique_crashes`, `unique_hangs`, `total_crashes`, +`slowest_exec_ms`, `edges_found`, `var_byte_count`, `havoc_expansion`. Their +definitions can be found in the addendum above. + +When using multiple fuzzer instances with StatsD, it is *strongly* recommended +to setup the flavor (AFL_STATSD_TAGS_FLAVOR) to match your StatsD server. This +will allow you to see individual fuzzer performance, detect bad ones, see the +progress of each strategy... \ No newline at end of file diff --git a/docs/best_practices.md b/docs/best_practices.md index 5d07dd14..979849f4 100644 --- a/docs/best_practices.md +++ b/docs/best_practices.md @@ -4,20 +4,26 @@ ### Targets - * [Fuzzing a binary-only target](#fuzzing-a-binary-only-target) - * [Fuzzing a GUI program](#fuzzing-a-gui-program) - * [Fuzzing a network service](#fuzzing-a-network-service) +* [Fuzzing a target with source code available](#fuzzing-a-target-with-source-code-available) +* [Fuzzing a binary-only target](#fuzzing-a-binary-only-target) +* [Fuzzing a GUI program](#fuzzing-a-gui-program) +* [Fuzzing a network service](#fuzzing-a-network-service) ### Improvements - * [Improving speed](#improving-speed) - * [Improving stability](#improving-stability) +* [Improving speed](#improving-speed) +* [Improving stability](#improving-stability) ## Targets +### Fuzzing a target with source code available + +To learn how to fuzz a target if source code is available, see [fuzzing_in_depth.md](fuzzing_in_depth.md). + ### Fuzzing a binary-only target -For a comprehensive guide, see [binaryonly_fuzzing.md](binaryonly_fuzzing.md). +For a comprehensive guide, see +[fuzzing_binary-only_targets.md](fuzzing_binary-only_targets.md). ### Fuzzing a GUI program @@ -48,7 +54,7 @@ to emulate the network. This is also much faster than the real network would be. See [utils/socket_fuzzing/](../utils/socket_fuzzing/). There is an outdated AFL++ branch that implements networking if you are -desperate though: [https://github.com/AFLplusplus/AFLplusplus/tree/networking](https://github.com/AFLplusplus/AFLplusplus/tree/networking) - +desperate though: [https://github.com/AFLplusplus/AFLplusplus/tree/networking](https://github.com/AFLplusplus/AFLplusplus/tree/networking) - however a better option is AFLnet ([https://github.com/aflnet/aflnet](https://github.com/aflnet/aflnet)) which allows you to define network state with different type of data packets. @@ -58,11 +64,11 @@ which allows you to define network state with different type of data packets. 1. Use [llvm_mode](../instrumentation/README.llvm.md): afl-clang-lto (llvm >= 11) or afl-clang-fast (llvm >= 9 recommended). 2. Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 speed increase). -3. Use the [AFL++ snapshot module](https://github.com/AFLplusplus/AFL-Snapshot-LKM) (x2 speed increase). +3. Instrument just what you are interested in, see [instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md). 4. If you do not use shmem persistent mode, use `AFL_TMPDIR` to put the input file directory on a tempfs location, see [env_variables.md](env_variables.md). 5. Improve Linux kernel performance: modify `/etc/default/grub`, set `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then `update-grub` and `reboot` (warning: makes the system less secure). 6. Running on an `ext2` filesystem with `noatime` mount option will be a bit faster than on any other journaling filesystem. -7. Use your cores! [fuzzing_expert.md:b) Using multiple cores](fuzzing_expert.md#b-using-multiple-cores). +7. Use your cores ([fuzzing_in_depth.md:3c) Using multiple cores](fuzzing_in_depth.md#c-using-multiple-cores))! ### Improving stability diff --git a/docs/beyond_crashes.md b/docs/beyond_crashes.md deleted file mode 100644 index 4836419c..00000000 --- a/docs/beyond_crashes.md +++ /dev/null @@ -1,23 +0,0 @@ -# Going beyond crashes - -Fuzzing is a wonderful and underutilized technique for discovering non-crashing -design and implementation errors, too. Quite a few interesting bugs have been -found by modifying the target programs to call abort() when say: - - - Two bignum libraries produce different outputs when given the same - fuzzer-generated input, - - - An image library produces different outputs when asked to decode the same - input image several times in a row, - - - A serialization / deserialization library fails to produce stable outputs - when iteratively serializing and deserializing fuzzer-supplied data, - - - A compression library produces an output inconsistent with the input file - when asked to compress and then decompress a particular blob. - -Implementing these or similar sanity checks usually takes very little time; -if you are the maintainer of a particular package, you can make this code -conditional with `#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` (a flag also -shared with libfuzzer and honggfuzz) or `#ifdef __AFL_COMPILER` (this one is -just for AFL). \ No newline at end of file diff --git a/docs/binaryonly_fuzzing.md b/docs/binaryonly_fuzzing.md deleted file mode 100644 index 2c0872cf..00000000 --- a/docs/binaryonly_fuzzing.md +++ /dev/null @@ -1,225 +0,0 @@ -# Fuzzing binary-only programs with AFL++ - - AFL++, libfuzzer and others are great if you have the source code, and - it allows for very fast and coverage guided fuzzing. - - However, if there is only the binary program and no source code available, - then standard `afl-fuzz -n` (non-instrumented mode) is not effective. - - The following is a description of how these binaries can be fuzzed with AFL++. - - -## TL;DR: - - qemu_mode in persistent mode is the fastest - if the stability is - high enough. Otherwise try retrowrite, afl-dyninst and if these - fail too then try standard qemu_mode with AFL_ENTRYPOINT to where you need it. - - If your target is a library use utils/afl_frida/. - - If your target is non-linux then use unicorn_mode/. - - -## QEMU - - Qemu is the "native" solution to the program. - It is available in the ./qemu_mode/ directory and once compiled it can - be accessed by the afl-fuzz -Q command line option. - It is the easiest to use alternative and even works for cross-platform binaries. - - The speed decrease is at about 50%. - However various options exist to increase the speed: - - using AFL_ENTRYPOINT to move the forkserver entry to a later basic block in - the binary (+5-10% speed) - - using persistent mode [qemu_mode/README.persistent.md](../qemu_mode/README.persistent.md) - this will result in 150-300% overall speed increase - so 3-8x the original - qemu_mode speed! - - using AFL_CODE_START/AFL_CODE_END to only instrument specific parts - - Note that there is also honggfuzz: [https://github.com/google/honggfuzz](https://github.com/google/honggfuzz) - which now has a qemu_mode, but its performance is just 1.5% ... - - As it is included in AFL++ this needs no URL. - - If you like to code a customized fuzzer without much work, we highly - recommend to check out our sister project libafl which will support QEMU - too: - [https://github.com/AFLplusplus/LibAFL](https://github.com/AFLplusplus/LibAFL) - - -## AFL FRIDA - - In frida_mode you can fuzz binary-only targets easily like with QEMU, - with the advantage that frida_mode also works on MacOS (both intel and M1). - - If you want to fuzz a binary-only library then you can fuzz it with - frida-gum via utils/afl_frida/, you will have to write a harness to - call the target function in the library, use afl-frida.c as a template. - - Both come with AFL++ so this needs no URL. - - You can also perform remote fuzzing with frida, e.g. if you want to fuzz - on iPhone or Android devices, for this you can use - [https://github.com/ttdennis/fpicker/](https://github.com/ttdennis/fpicker/) - as an intermediate that uses AFL++ for fuzzing. - - If you like to code a customized fuzzer without much work, we highly - recommend to check out our sister project libafl which supports Frida too: - [https://github.com/AFLplusplus/LibAFL](https://github.com/AFLplusplus/LibAFL) - Working examples already exist :-) - - -## WINE+QEMU - - Wine mode can run Win32 PE binaries with the QEMU instrumentation. - It needs Wine, python3 and the pefile python package installed. - - As it is included in AFL++ this needs no URL. - - -## UNICORN - - Unicorn is a fork of QEMU. The instrumentation is, therefore, very similar. - In contrast to QEMU, Unicorn does not offer a full system or even userland - emulation. Runtime environment and/or loaders have to be written from scratch, - if needed. On top, block chaining has been removed. This means the speed boost - introduced in the patched QEMU Mode of AFL++ cannot simply be ported over to - Unicorn. For further information, check out [unicorn_mode/README.md](../unicorn_mode/README.md). - - As it is included in AFL++ this needs no URL. - - -## AFL UNTRACER - - If you want to fuzz a binary-only shared library then you can fuzz it with - utils/afl_untracer/, use afl-untracer.c as a template. - It is slower than AFL FRIDA (see above). - - -## ZAFL - ZAFL is a static rewriting platform supporting x86-64 C/C++, stripped/unstripped, - and PIE/non-PIE binaries. Beyond conventional instrumentation, ZAFL's API enables - transformation passes (e.g., laf-Intel, context sensitivity, InsTrim, etc.). - - Its baseline instrumentation speed typically averages 90-95% of afl-clang-fast's. - - [https://git.zephyr-software.com/opensrc/zafl](https://git.zephyr-software.com/opensrc/zafl) - - -## DYNINST - - Dyninst is a binary instrumentation framework similar to Pintool and - Dynamorio (see far below). However whereas Pintool and Dynamorio work at - runtime, dyninst instruments the target at load time, and then let it run - - or save the binary with the changes. - This is great for some things, e.g. fuzzing, and not so effective for others, - e.g. malware analysis. - - So what we can do with dyninst is taking every basic block, and put afl's - instrumention code in there - and then save the binary. - Afterwards we can just fuzz the newly saved target binary with afl-fuzz. - Sounds great? It is. The issue though - it is a non-trivial problem to - insert instructions, which change addresses in the process space, so that - everything is still working afterwards. Hence more often than not binaries - crash when they are run. - - The speed decrease is about 15-35%, depending on the optimization options - used with afl-dyninst. - - [https://github.com/vanhauser-thc/afl-dyninst](https://github.com/vanhauser-thc/afl-dyninst) - - -## RETROWRITE - - If you have an x86/x86_64 binary that still has its symbols, is compiled - with position independant code (PIC/PIE) and does not use most of the C++ - features then the retrowrite solution might be for you. - It decompiles to ASM files which can then be instrumented with afl-gcc. - - It is at about 80-85% performance. - - [https://github.com/HexHive/retrowrite](https://github.com/HexHive/retrowrite) - - -## MCSEMA - - Theoretically you can also decompile to llvm IR with mcsema, and then - use llvm_mode to instrument the binary. - Good luck with that. - - [https://github.com/lifting-bits/mcsema](https://github.com/lifting-bits/mcsema) - - -## INTEL-PT - - If you have a newer Intel CPU, you can make use of Intels processor trace. - The big issue with Intel's PT is the small buffer size and the complex - encoding of the debug information collected through PT. - This makes the decoding very CPU intensive and hence slow. - As a result, the overall speed decrease is about 70-90% (depending on - the implementation and other factors). - - There are two AFL intel-pt implementations: - - 1. [https://github.com/junxzm1990/afl-pt](https://github.com/junxzm1990/afl-pt) - => this needs Ubuntu 14.04.05 without any updates and the 4.4 kernel. - - 2. [https://github.com/hunter-ht-2018/ptfuzzer](https://github.com/hunter-ht-2018/ptfuzzer) - => this needs a 4.14 or 4.15 kernel. the "nopti" kernel boot option must - be used. This one is faster than the other. - - Note that there is also honggfuzz: https://github.com/google/honggfuzz - But its IPT performance is just 6%! - - -## CORESIGHT - - Coresight is ARM's answer to Intel's PT. - With afl++ v3.15 there is a coresight tracer implementation available in - `coresight_mode/` which is faster than QEMU, however can not run in parallel. - Currently only one process can be traced, it is WIP. - - -## PIN & DYNAMORIO - - Pintool and Dynamorio are dynamic instrumentation engines, and they can be - used for getting basic block information at runtime. - Pintool is only available for Intel x32/x64 on Linux, Mac OS and Windows, - whereas Dynamorio is additionally available for ARM and AARCH64. - Dynamorio is also 10x faster than Pintool. - - The big issue with Dynamorio (and therefore Pintool too) is speed. - Dynamorio has a speed decrease of 98-99% - Pintool has a speed decrease of 99.5% - - Hence Dynamorio is the option to go for if everything else fails, and Pintool - only if Dynamorio fails too. - - Dynamorio solutions: - * [https://github.com/vanhauser-thc/afl-dynamorio](https://github.com/vanhauser-thc/afl-dynamorio) - * [https://github.com/mxmssh/drAFL](https://github.com/mxmssh/drAFL) - * [https://github.com/googleprojectzero/winafl/](https://github.com/googleprojectzero/winafl/) <= very good but windows only - - Pintool solutions: - * [https://github.com/vanhauser-thc/afl-pin](https://github.com/vanhauser-thc/afl-pin) - * [https://github.com/mothran/aflpin](https://github.com/mothran/aflpin) - * [https://github.com/spinpx/afl_pin_mode](https://github.com/spinpx/afl_pin_mode) <= only old Pintool version supported - - -## Non-AFL solutions - - There are many binary-only fuzzing frameworks. - Some are great for CTFs but don't work with large binaries, others are very - slow but have good path discovery, some are very hard to set-up ... - - * QSYM: [https://github.com/sslab-gatech/qsym](https://github.com/sslab-gatech/qsym) - * Manticore: [https://github.com/trailofbits/manticore](https://github.com/trailofbits/manticore) - * S2E: [https://github.com/S2E](https://github.com/S2E) - * Tinyinst: [https://github.com/googleprojectzero/TinyInst](https://github.com/googleprojectzero/TinyInst) (Mac/Windows only) - * Jackalope: [https://github.com/googleprojectzero/Jackalope](https://github.com/googleprojectzero/Jackalope) - * ... please send me any missing that are good - - -## Closing words - - That's it! News, corrections, updates? Send an email to vh@thc.org diff --git a/docs/branches.md b/docs/branches.md deleted file mode 100644 index ae147b08..00000000 --- a/docs/branches.md +++ /dev/null @@ -1,11 +0,0 @@ -# Branches - -The following branches exist: - -* [release](https://github.com/AFLplusplus/AFLplusplus/tree/release): the latest release -* [stable/trunk](https://github.com/AFLplusplus/AFLplusplus/): stable state of AFL++ - it is synced from dev from time to time when we are satisfied with its stability -* [dev](https://github.com/AFLplusplus/AFLplusplus/tree/dev): development state of AFL++ - bleeding edge and you might catch a checkout which does not compile or has a bug. *We only accept PRs in dev!!* -* (any other): experimental branches to work on specific features or testing new functionality or changes. - -For releases, please see the [Releases](https://github.com/AFLplusplus/AFLplusplus/releases) tab. -Also take a look at the list of [important changes in AFL++](important_changes.md). \ No newline at end of file diff --git a/docs/choosing_testcases.md b/docs/choosing_testcases.md deleted file mode 100644 index 25002929..00000000 --- a/docs/choosing_testcases.md +++ /dev/null @@ -1,19 +0,0 @@ -# Choosing initial test cases - -To operate correctly, the fuzzer requires one or more starting file that -contains a good example of the input data normally expected by the targeted -application. There are two basic rules: - - - Keep the files small. Under 1 kB is ideal, although not strictly necessary. - For a discussion of why size matters, see [perf_tips.md](perf_tips.md). - - - Use multiple test cases only if they are functionally different from - each other. There is no point in using fifty different vacation photos - to fuzz an image library. - -You can find many good examples of starting files in the testcases/ subdirectory -that comes with this tool. - -PS. If a large corpus of data is available for screening, you may want to use -the afl-cmin utility to identify a subset of functionally distinct files that -exercise different code paths in the target binary. \ No newline at end of file diff --git a/docs/ci_fuzzing.md b/docs/ci_fuzzing.md deleted file mode 100644 index 8d1a2f99..00000000 --- a/docs/ci_fuzzing.md +++ /dev/null @@ -1,29 +0,0 @@ -# CI Fuzzing - -Some notes on CI Fuzzing - this fuzzing is different to normal fuzzing campaigns as these are much shorter runnings. - -1. Always: - * LTO has a much longer compile time which is diametrical to short fuzzing - hence use afl-clang-fast instead. - * If you compile with CMPLOG then you can save fuzzing time and reuse that compiled target for both the -c option and the main fuzz target. - This will impact the speed by ~15% though. - * `AFL_FAST_CAL` - Enable fast calibration, this halfs the time the saturated corpus needs to be loaded. - * `AFL_CMPLOG_ONLY_NEW` - only perform cmplog on new found paths, not the initial corpus as this very likely has been done for them already. - * Keep the generated corpus, use afl-cmin and reuse it every time! - -2. Additionally randomize the AFL++ compilation options, e.g. - * 40% for `AFL_LLVM_CMPLOG` - * 10% for `AFL_LLVM_LAF_ALL` - -3. Also randomize the afl-fuzz runtime options, e.g. - * 65% for `AFL_DISABLE_TRIM` - * 50% use a dictionary generated by `AFL_LLVM_DICT2FILE` - * 40% use MOpt (`-L 0`) - * 40% for `AFL_EXPAND_HAVOC_NOW` - * 20% for old queue processing (`-Z`) - * for CMPLOG targets, 60% for `-l 2`, 40% for `-l 3` - -4. Do *not* run any `-M` modes, just running `-S` modes is better for CI fuzzing. -`-M` enables old queue handling etc. which is good for a fuzzing campaign but not good for short CI runs. - -How this can look like can e.g. be seen at AFL++'s setup in Google's [oss-fuzz](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/compile_afl) -and [clusterfuzz](https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/bot/fuzzers/afl/launcher.py). diff --git a/docs/common_sense_risks.md b/docs/common_sense_risks.md deleted file mode 100644 index a8d68d7a..00000000 --- a/docs/common_sense_risks.md +++ /dev/null @@ -1,36 +0,0 @@ -# Common sense risks - -Please keep in mind that, similarly to many other computationally-intensive -tasks, fuzzing may put a strain on your hardware and on the OS. In particular: - - - Your CPU will run hot and will need adequate cooling. In most cases, if - cooling is insufficient or stops working properly, CPU speeds will be - automatically throttled. That said, especially when fuzzing on less - suitable hardware (laptops, smartphones, etc), it's not entirely impossible - for something to blow up. - - - Targeted programs may end up erratically grabbing gigabytes of memory or - filling up disk space with junk files. AFL++ tries to enforce basic memory - limits, but can't prevent each and every possible mishap. The bottom line - is that you shouldn't be fuzzing on systems where the prospect of data loss - is not an acceptable risk. - - - Fuzzing involves billions of reads and writes to the filesystem. On modern - systems, this will be usually heavily cached, resulting in fairly modest - "physical" I/O - but there are many factors that may alter this equation. - It is your responsibility to monitor for potential trouble; with very heavy - I/O, the lifespan of many HDDs and SSDs may be reduced. - - A good way to monitor disk I/O on Linux is the 'iostat' command: - -```shell - $ iostat -d 3 -x -k [...optional disk ID...] -``` - - Using the `AFL_TMPDIR` environment variable and a RAM-disk you can have the - heavy writing done in RAM to prevent the aforementioned wear and tear. For - example the following line will run a Docker container with all this preset: - - ```shell - # docker run -ti --mount type=tmpfs,destination=/ramdisk -e AFL_TMPDIR=/ramdisk aflplusplus/aflplusplus - ``` \ No newline at end of file diff --git a/docs/custom_mutators.md b/docs/custom_mutators.md index 8b5a4068..b1dfd309 100644 --- a/docs/custom_mutators.md +++ b/docs/custom_mutators.md @@ -127,9 +127,9 @@ def deinit(): # optional for Python - `describe` (optional): - When this function is called, it shall describe the current testcase, + When this function is called, it shall describe the current test case, generated by the last mutation. This will be called, for example, - to name the written testcase file after a crash occurred. + to name the written test case file after a crash occurred. Using it can help to reproduce crashing mutations. - `havoc_mutation` and `havoc_mutation_probability` (optional): @@ -224,7 +224,7 @@ Optionally, the following environment variables are supported: - `AFL_CUSTOM_MUTATOR_ONLY` - Disable all other mutation stages. This can prevent broken testcases + Disable all other mutation stages. This can prevent broken test cases (those that your Python module can't work with anymore) to fill up your queue. Best combined with a custom trimming routine (see below) because trimming can cause the same test breakage like havoc and splice. diff --git a/docs/env_variables.md b/docs/env_variables.md index cbc63032..cce9ff56 100644 --- a/docs/env_variables.md +++ b/docs/env_variables.md @@ -143,7 +143,7 @@ Available options: - CLANG - outdated clang instrumentation - CLASSIC - classic AFL (map[cur_loc ^ prev_loc >> 1]++) (default) - You can also specify CTX and/or NGRAM, seperate the options with a comma "," + You can also specify CTX and/or NGRAM, separate the options with a comma "," then, e.g.: `AFL_LLVM_INSTRUMENT=CLASSIC,CTX,NGRAM-4` Note: It is actually not a good idea to use both CTX and NGRAM. :) @@ -303,8 +303,9 @@ checks or alter some of the more exotic semantics of the tool: exit soon after the first crash is found. - `AFL_CMPLOG_ONLY_NEW` will only perform the expensive cmplog feature for - newly found testcases and not for testcases that are loaded on startup (`-i - in`). This is an important feature to set when resuming a fuzzing session. + newly found test cases and not for test cases that are loaded on startup + (`-i in`). This is an important feature to set when resuming a fuzzing + session. - Setting `AFL_CRASH_EXITCODE` sets the exit code AFL treats as crash. For example, if `AFL_CRASH_EXITCODE='-1'` is set, each input resulting in a `-1` @@ -444,8 +445,8 @@ checks or alter some of the more exotic semantics of the tool: - If you are using persistent mode (you should, see [instrumentation/README.persistent_mode.md](../instrumentation/README.persistent_mode.md)), - some targets keep inherent state due which a detected crash testcase does - not crash the target again when the testcase is given. To be able to still + some targets keep inherent state due which a detected crash test case does + not crash the target again when the test case is given. To be able to still re-trigger these crashes, you can use the `AFL_PERSISTENT_RECORD` variable with a value of how many previous fuzz cases to keep prio a crash. If set to e.g. 10, then the 9 previous inputs are written to out/default/crashes as @@ -523,23 +524,23 @@ checks or alter some of the more exotic semantics of the tool: The QEMU wrapper used to instrument binary-only code supports several settings: - Setting `AFL_COMPCOV_LEVEL` enables the CompareCoverage tracing of all cmp - and sub in x86 and x86_64 and memory comparions functions (e.g. strcmp, + and sub in x86 and x86_64 and memory comparison functions (e.g., strcmp, memcmp, ...) when libcompcov is preloaded using `AFL_PRELOAD`. More info at [qemu_mode/libcompcov/README.md](../qemu_mode/libcompcov/README.md). There are two levels at the moment, `AFL_COMPCOV_LEVEL=1` that instruments only comparisons with immediate values / read-only memory and - `AFL_COMPCOV_LEVEL=2` that instruments all the comparions. Level 2 is more + `AFL_COMPCOV_LEVEL=2` that instruments all the comparisons. Level 2 is more accurate but may need a larger shared memory. - - `AFL_DEBUG` will print the found entrypoint for the binary to stderr. Use - this if you are unsure if the entrypoint might be wrong - but use it + - `AFL_DEBUG` will print the found entry point for the binary to stderr. Use + this if you are unsure if the entry point might be wrong - but use it directly, e.g. `afl-qemu-trace ./program`. - - `AFL_ENTRYPOINT` allows you to specify a specific entrypoint into the binary - (this can be very good for the performance!). The entrypoint is specified as - hex address, e.g. `0x4004110`. Note that the address must be the address of - a basic block. + - `AFL_ENTRYPOINT` allows you to specify a specific entry point into the + binary (this can be very good for the performance!). The entry point is + specified as hex address, e.g. `0x4004110`. Note that the address must be + the address of a basic block. - Setting `AFL_INST_LIBS` causes the translator to also instrument the code inside any dynamically linked libraries (notably including glibc). diff --git a/docs/features.md b/docs/features.md index f44e32ff..35a869a9 100644 --- a/docs/features.md +++ b/docs/features.md @@ -1,49 +1,61 @@ # Important features of AFL++ - AFL++ supports llvm from 3.8 up to version 12, very fast binary fuzzing with QEMU 5.1 - with laf-intel and redqueen, frida mode, unicorn mode, gcc plugin, full *BSD, - Mac OS, Solaris and Android support and much, much, much more. +AFL++ supports llvm from 3.8 up to version 12, very fast binary fuzzing with +QEMU 5.1 with laf-intel and redqueen, frida mode, unicorn mode, gcc plugin, full +*BSD, Mac OS, Solaris and Android support and much, much, much more. - | Feature/Instrumentation | afl-gcc | llvm | gcc_plugin | frida_mode(9) | qemu_mode(10) |unicorn_mode(10) |coresight_mode(11)| - | -------------------------|:-------:|:---------:|:----------:|:----------------:|:----------------:|:----------------:|:----------------:| - | Threadsafe counters | | x(3) | | | | | | - | NeverZero | x86[_64]| x(1) | x | x | x | x | | - | Persistent Mode | | x | x | x86[_64]/arm64 | x86[_64]/arm[64] | x | | - | LAF-Intel / CompCov | | x | | | x86[_64]/arm[64] | x86[_64]/arm[64] | | - | CmpLog | | x | | x86[_64]/arm64 | x86[_64]/arm[64] | | | - | Selective Instrumentation| | x | x | x | x | | | - | Non-Colliding Coverage | | x(4) | | | (x)(5) | | | - | Ngram prev_loc Coverage | | x(6) | | | | | | - | Context Coverage | | x(6) | | | | | | - | Auto Dictionary | | x(7) | | | | | | - | Snapshot LKM Support | | (x)(8) | (x)(8) | | (x)(5) | | | - | Shared Memory Testcases | | x | x | x86[_64]/arm64 | x | x | | +| Feature/Instrumentation | afl-gcc | llvm | gcc_plugin | frida_mode(9) | qemu_mode(10) |unicorn_mode(10) |coresight_mode(11)| +| -------------------------|:-------:|:---------:|:----------:|:----------------:|:----------------:|:----------------:|:----------------:| +| Threadsafe counters | | x(3) | | | | | | +| NeverZero | x86[_64]| x(1) | x | x | x | x | | +| Persistent Mode | | x | x | x86[_64]/arm64 | x86[_64]/arm[64] | x | | +| LAF-Intel / CompCov | | x | | | x86[_64]/arm[64] | x86[_64]/arm[64] | | +| CmpLog | | x | | x86[_64]/arm64 | x86[_64]/arm[64] | | | +| Selective Instrumentation| | x | x | x | x | | | +| Non-Colliding Coverage | | x(4) | | | (x)(5) | | | +| Ngram prev_loc Coverage | | x(6) | | | | | | +| Context Coverage | | x(6) | | | | | | +| Auto Dictionary | | x(7) | | | | | | +| Snapshot LKM Support | | (x)(8) | (x)(8) | | (x)(5) | | | +| Shared Memory Test cases | | x | x | x86[_64]/arm64 | x | x | | - 1. default for LLVM >= 9.0, env var for older version due an efficiency bug in previous llvm versions - 2. GCC creates non-performant code, hence it is disabled in gcc_plugin - 3. with `AFL_LLVM_THREADSAFE_INST`, disables NeverZero - 4. with pcguard mode and LTO mode for LLVM 11 and newer - 5. upcoming, development in the branch - 6. not compatible with LTO instrumentation and needs at least LLVM v4.1 - 7. automatic in LTO mode with LLVM 11 and newer, an extra pass for all LLVM versions that write to a file to use with afl-fuzz' `-x` - 8. the snapshot LKM is currently unmaintained due to too many kernel changes coming too fast :-( - 9. frida mode is supported on Linux and MacOS for Intel and ARM - 10. QEMU/Unicorn is only supported on Linux - 11. Coresight mode is only available on AARCH64 Linux with a CPU with Coresight extension +1. default for LLVM >= 9.0, env var for older version due an efficiency bug in + previous llvm versions +2. GCC creates non-performant code, hence it is disabled in gcc_plugin +3. with `AFL_LLVM_THREADSAFE_INST`, disables NeverZero +4. with pcguard mode and LTO mode for LLVM 11 and newer +5. upcoming, development in the branch +6. not compatible with LTO instrumentation and needs at least LLVM v4.1 +7. automatic in LTO mode with LLVM 11 and newer, an extra pass for all LLVM + versions that write to a file to use with afl-fuzz' `-x` +8. the snapshot LKM is currently unmaintained due to too many kernel changes + coming too fast :-( +9. frida mode is supported on Linux and MacOS for Intel and ARM +10. QEMU/Unicorn is only supported on Linux +11. Coresight mode is only available on AARCH64 Linux with a CPU with Coresight + extension - Among others, the following features and patches have been integrated: +Among others, the following features and patches have been integrated: - * NeverZero patch for afl-gcc, instrumentation, qemu_mode and unicorn_mode which prevents a wrapping map value to zero, increases coverage - * Persistent mode, deferred forkserver and in-memory fuzzing for qemu_mode - * Unicorn mode which allows fuzzing of binaries from completely different platforms (integration provided by domenukk) - * The new CmpLog instrumentation for LLVM and QEMU inspired by [Redqueen](https://www.syssec.ruhr-uni-bochum.de/media/emma/veroeffentlichungen/2018/12/17/NDSS19-Redqueen.pdf) - * Win32 PE binary-only fuzzing with QEMU and Wine - * AFLfast's power schedules by Marcel Böhme: [https://github.com/mboehme/aflfast](https://github.com/mboehme/aflfast) - * The MOpt mutator: [https://github.com/puppet-meteor/MOpt-AFL](https://github.com/puppet-meteor/MOpt-AFL) - * LLVM mode Ngram coverage by Adrian Herrera [https://github.com/adrianherrera/afl-ngram-pass](https://github.com/adrianherrera/afl-ngram-pass) - * LAF-Intel/CompCov support for instrumentation, qemu_mode and unicorn_mode (with enhanced capabilities) - * Radamsa and honggfuzz mutators (as custom mutators). - * QBDI mode to fuzz android native libraries via Quarkslab's [QBDI](https://github.com/QBDI/QBDI) framework - * Frida and ptrace mode to fuzz binary-only libraries, etc. +* NeverZero patch for afl-gcc, instrumentation, qemu_mode and unicorn_mode which + prevents a wrapping map value to zero, increases coverage +* Persistent mode, deferred forkserver and in-memory fuzzing for qemu_mode +* Unicorn mode which allows fuzzing of binaries from completely different + platforms (integration provided by domenukk) +* The new CmpLog instrumentation for LLVM and QEMU inspired by + [Redqueen](https://www.syssec.ruhr-uni-bochum.de/media/emma/veroeffentlichungen/2018/12/17/NDSS19-Redqueen.pdf) +* Win32 PE binary-only fuzzing with QEMU and Wine +* AFLfast's power schedules by Marcel Böhme: + [https://github.com/mboehme/aflfast](https://github.com/mboehme/aflfast) +* The MOpt mutator: + [https://github.com/puppet-meteor/MOpt-AFL](https://github.com/puppet-meteor/MOpt-AFL) +* LLVM mode Ngram coverage by Adrian Herrera + [https://github.com/adrianherrera/afl-ngram-pass](https://github.com/adrianherrera/afl-ngram-pass) +* LAF-Intel/CompCov support for instrumentation, qemu_mode and unicorn_mode + (with enhanced capabilities) +* Radamsa and honggfuzz mutators (as custom mutators). +* QBDI mode to fuzz android native libraries via Quarkslab's + [QBDI](https://github.com/QBDI/QBDI) framework +* Frida and ptrace mode to fuzz binary-only libraries, etc. - So all in all this is the best-of AFL that is out there :-) \ No newline at end of file +So all in all this is the best-of AFL that is out there :-) \ No newline at end of file diff --git a/docs/fuzzing_binary-only_targets.md b/docs/fuzzing_binary-only_targets.md index ea262f6e..290c9bec 100644 --- a/docs/fuzzing_binary-only_targets.md +++ b/docs/fuzzing_binary-only_targets.md @@ -1,83 +1,293 @@ # Fuzzing binary-only targets -When source code is *NOT* available, AFL++ offers various support for fast, -on-the-fly instrumentation of black-box binaries. +AFL++, libfuzzer, and other fuzzers are great if you have the source code of the +target. This allows for very fast and coverage guided fuzzing. -If you do not have to use Unicorn the following setup is recommended to use -qemu_mode: - * run 1 afl-fuzz -Q instance with CMPLOG (`-c 0` + `AFL_COMPCOV_LEVEL=2`) - * run 1 afl-fuzz -Q instance with QASAN (`AFL_USE_QASAN=1`) - * run 1 afl-fuzz -Q instance with LAF (`AFL_PRELOAD=libcmpcov.so` + `AFL_COMPCOV_LEVEL=2`) -Alternatively you can use frida_mode, just switch `-Q` with `-O` and remove the -LAF instance. +However, if there is only the binary program and no source code available, then +standard `afl-fuzz -n` (non-instrumented mode) is not effective. -Then run as many instances as you have cores left with either -Q mode or - better - -use a binary rewriter like afl-dyninst, retrowrite, zafl, etc. +For fast, on-the-fly instrumentation of black-box binaries, AFL++ still offers +various support. The following is a description of how these binaries can be +fuzzed with AFL++. -For Qemu and Frida mode, check out the persistent mode, it gives a huge speed -improvement if it is possible to use. +## TL;DR: -### QEMU +Qemu_mode in persistent mode is the fastest - if the stability is high enough. +Otherwise, try RetroWrite, Dyninst, and if these fail, too, then try standard +qemu_mode with AFL_ENTRYPOINT to where you need it. -For linux programs and its libraries this is accomplished with a version of -QEMU running in the lesser-known "user space emulation" mode. -QEMU is a project separate from AFL, but you can conveniently build the -feature by doing: +If your target is a library, then use frida_mode. + +If your target is non-linux, then use unicorn_mode. + +## Fuzzing binary-only targets with AFL++ +### Qemu_mode + +Qemu_mode is the "native" solution to the program. It is available in the +./qemu_mode/ directory and, once compiled, it can be accessed by the afl-fuzz -Q +command line option. It is the easiest to use alternative and even works for +cross-platform binaries. + +For linux programs and its libraries, this is accomplished with a version of +QEMU running in the lesser-known "user space emulation" mode. QEMU is a project +separate from AFL++, but you can conveniently build the feature by doing: ```shell cd qemu_mode ./build_qemu_support.sh ``` -For additional instructions and caveats, see [qemu_mode/README.md](../qemu_mode/README.md). -If possible you should use the persistent mode, see [qemu_mode/README.persistent.md](../qemu_mode/README.persistent.md). -The mode is approximately 2-5x slower than compile-time instrumentation, and is -less conducive to parallelization. +The following setup to use qemu_mode is recommended: +* run 1 afl-fuzz -Q instance with CMPLOG (`-c 0` + `AFL_COMPCOV_LEVEL=2`) +* run 1 afl-fuzz -Q instance with QASAN (`AFL_USE_QASAN=1`) +* run 1 afl-fuzz -Q instance with LAF (`AFL_PRELOAD=libcmpcov.so` + + `AFL_COMPCOV_LEVEL=2`), alternatively you can use frida_mode, just switch `-Q` + with `-O` and remove the LAF instance + +Then run as many instances as you have cores left with either -Q mode or - even +better - use a binary rewriter like Dyninst, RetroWrite, ZAFL, etc. + +If [afl-dyninst](https://github.com/vanhauser-thc/afl-dyninst) works for your +binary, then you can use afl-fuzz normally and it will have twice the speed +compared to qemu_mode (but slower than qemu persistent mode). Note that several +other binary rewriters exist, all with their advantages and caveats. + +The speed decrease of qemu_mode is at about 50%. However, various options exist +to increase the speed: +- using AFL_ENTRYPOINT to move the forkserver entry to a later basic block in + the binary (+5-10% speed) +- using persistent mode + [qemu_mode/README.persistent.md](../qemu_mode/README.persistent.md) this will + result in a 150-300% overall speed increase - so 3-8x the original qemu_mode + speed! +- using AFL_CODE_START/AFL_CODE_END to only instrument specific parts + +For additional instructions and caveats, see +[qemu_mode/README.md](../qemu_mode/README.md). If possible, you should use the +persistent mode, see +[qemu_mode/README.persistent.md](../qemu_mode/README.persistent.md). The mode is +approximately 2-5x slower than compile-time instrumentation, and is less +conducive to parallelization. -If [afl-dyninst](https://github.com/vanhauser-thc/afl-dyninst) works for -your binary, then you can use afl-fuzz normally and it will have twice -the speed compared to qemu_mode (but slower than qemu persistent mode). -Note that several other binary rewriters exist, all with their advantages and -caveats. +Note that there is also honggfuzz: +[https://github.com/google/honggfuzz](https://github.com/google/honggfuzz) which +now has a qemu_mode, but its performance is just 1.5% ... -### Frida +If you like to code a customized fuzzer without much work, we highly recommend +to check out our sister project libafl which supports QEMU, too: +[https://github.com/AFLplusplus/LibAFL](https://github.com/AFLplusplus/LibAFL) -Frida mode is sometimes faster and sometimes slower than Qemu mode. -It is also newer, lacks COMPCOV, but supports MacOS. +### WINE+QEMU + +Wine mode can run Win32 PE binaries with the QEMU instrumentation. It needs +Wine, python3, and the pefile python package installed. + +It is included in AFL++. + +For more information, see [qemu_mode/README.wine.md](../qemu_mode/README.wine.md). + +### Frida_mode + +In frida_mode, you can fuzz binary-only targets as easily as with QEMU. +Frida_mode is sometimes faster and sometimes slower than Qemu_mode. It is also +newer, lacks COMPCOV, and has the advantage that it works on MacOS (both intel +and M1). + +To build frida_mode: ```shell cd frida_mode make ``` -For additional instructions and caveats, see [frida_mode/README.md](../frida_mode/README.md). -If possible you should use the persistent mode, see [qemu_frida/README.md](../qemu_frida/README.md). -The mode is approximately 2-5x slower than compile-time instrumentation, and is -less conducive to parallelization. +For additional instructions and caveats, see +[frida_mode/README.md](../frida_mode/README.md). + +If possible, you should use the persistent mode, see +[qemu_frida/README.md](../qemu_frida/README.md). The mode is approximately 2-5x +slower than compile-time instrumentation, and is less conducive to +parallelization. But for binary-only fuzzing, it gives a huge speed improvement +if it is possible to use. + +If you want to fuzz a binary-only library, then you can fuzz it with frida-gum +via frida_mode/. You will have to write a harness to call the target function in +the library, use afl-frida.c as a template. + +You can also perform remote fuzzing with frida, e.g. if you want to fuzz on +iPhone or Android devices, for this you can use +[https://github.com/ttdennis/fpicker/](https://github.com/ttdennis/fpicker/) as +an intermediate that uses AFL++ for fuzzing. + +If you like to code a customized fuzzer without much work, we highly recommend +to check out our sister project libafl which supports Frida, too: +[https://github.com/AFLplusplus/LibAFL](https://github.com/AFLplusplus/LibAFL). +Working examples already exist :-) ### Unicorn -For non-Linux binaries you can use AFL++'s unicorn mode which can emulate +Unicorn is a fork of QEMU. The instrumentation is, therefore, very similar. In +contrast to QEMU, Unicorn does not offer a full system or even userland +emulation. Runtime environment and/or loaders have to be written from scratch, +if needed. On top, block chaining has been removed. This means the speed boost +introduced in the patched QEMU Mode of AFL++ cannot simply be ported over to +Unicorn. + +For non-Linux binaries, you can use AFL++'s unicorn_mode which can emulate anything you want - for the price of speed and user written scripts. -See [unicorn_mode/README.md](../unicorn_mode/README.md). -It can be easily built by: +To build unicorn_mode: + ```shell cd unicorn_mode ./build_unicorn_support.sh ``` +For further information, check out +[unicorn_mode/README.md](../unicorn_mode/README.md). + ### Shared libraries -If the goal is to fuzz a dynamic library then there are two options available. -For both you need to write a small harness that loads and calls the library. -Then you fuzz this with either frida_mode or qemu_mode, and either use +If the goal is to fuzz a dynamic library, then there are two options available. +For both, you need to write a small harness that loads and calls the library. +Then you fuzz this with either frida_mode or qemu_mode and either use `AFL_INST_LIBS=1` or `AFL_QEMU/FRIDA_INST_RANGES`. -Another, less precise and slower option is using ptrace with debugger interrupt -instrumentation: [utils/afl_untracer/README.md](../utils/afl_untracer/README.md). +Another, less precise and slower option is to fuzz it with utils/afl_untracer/ +and use afl-untracer.c as a template. It is slower than frida_mode. + +For more information, see +[utils/afl_untracer/README.md](../utils/afl_untracer/README.md). + +### Coresight + +Coresight is ARM's answer to Intel's PT. With AFL++ v3.15, there is a coresight +tracer implementation available in `coresight_mode/` which is faster than QEMU, +however, cannot run in parallel. Currently, only one process can be traced, it +is WIP. + +Fore more information, see +[coresight_mode/README.md](../coresight_mode/README.md). + +## Binary rewriters + +An alternative solution are binary rewriters. They are faster then the solutions native to AFL++ but don't always work. + +### ZAFL +ZAFL is a static rewriting platform supporting x86-64 C/C++, +stripped/unstripped, and PIE/non-PIE binaries. Beyond conventional +instrumentation, ZAFL's API enables transformation passes (e.g., laf-Intel, +context sensitivity, InsTrim, etc.). + +Its baseline instrumentation speed typically averages 90-95% of +afl-clang-fast's. + +[https://git.zephyr-software.com/opensrc/zafl](https://git.zephyr-software.com/opensrc/zafl) + +### RetroWrite + +If you have an x86/x86_64 binary that still has its symbols, is compiled with +position independent code (PIC/PIE), and does not use most of the C++ features, +then the RetroWrite solution might be for you. It decompiles to ASM files which +can then be instrumented with afl-gcc. + +It is at about 80-85% performance. + +[https://github.com/HexHive/retrowrite](https://github.com/HexHive/retrowrite) + +### Dyninst + +Dyninst is a binary instrumentation framework similar to Pintool and DynamoRIO. +However, whereas Pintool and DynamoRIO work at runtime, Dyninst instruments the +target at load time and then let it run - or save the binary with the changes. +This is great for some things, e.g. fuzzing, and not so effective for others, +e.g. malware analysis. + +So, what we can do with Dyninst is taking every basic block and put AFL++'s +instrumentation code in there - and then save the binary. Afterwards, we can +just fuzz the newly saved target binary with afl-fuzz. Sounds great? It is. The +issue though - it is a non-trivial problem to insert instructions, which change +addresses in the process space, so that everything is still working afterwards. +Hence, more often than not binaries crash when they are run. + +The speed decrease is about 15-35%, depending on the optimization options used +with afl-dyninst. + +[https://github.com/vanhauser-thc/afl-dyninst](https://github.com/vanhauser-thc/afl-dyninst) + +### Mcsema + +Theoretically, you can also decompile to llvm IR with mcsema, and then use +llvm_mode to instrument the binary. Good luck with that. + +[https://github.com/lifting-bits/mcsema](https://github.com/lifting-bits/mcsema) + +## Binary tracers + +### Pintool & DynamoRIO + +Pintool and DynamoRIO are dynamic instrumentation engines. They can be used for +getting basic block information at runtime. Pintool is only available for Intel +x32/x64 on Linux, Mac OS, and Windows, whereas DynamoRIO is additionally +available for ARM and AARCH64. DynamoRIO is also 10x faster than Pintool. + +The big issue with DynamoRIO (and therefore Pintool, too) is speed. DynamoRIO +has a speed decrease of 98-99%, Pintool has a speed decrease of 99.5%. + +Hence, DynamoRIO is the option to go for if everything else fails and Pintool +only if DynamoRIO fails, too. + +DynamoRIO solutions: +* [https://github.com/vanhauser-thc/afl-dynamorio](https://github.com/vanhauser-thc/afl-dynamorio) +* [https://github.com/mxmssh/drAFL](https://github.com/mxmssh/drAFL) +* [https://github.com/googleprojectzero/winafl/](https://github.com/googleprojectzero/winafl/) + <= very good but windows only + +Pintool solutions: +* [https://github.com/vanhauser-thc/afl-pin](https://github.com/vanhauser-thc/afl-pin) +* [https://github.com/mothran/aflpin](https://github.com/mothran/aflpin) +* [https://github.com/spinpx/afl_pin_mode](https://github.com/spinpx/afl_pin_mode) + <= only old Pintool version supported + +### Intel PT + +If you have a newer Intel CPU, you can make use of Intel's processor trace. The +big issue with Intel's PT is the small buffer size and the complex encoding of +the debug information collected through PT. This makes the decoding very CPU +intensive and hence slow. As a result, the overall speed decrease is about +70-90% (depending on the implementation and other factors). + +There are two AFL intel-pt implementations: + +1. [https://github.com/junxzm1990/afl-pt](https://github.com/junxzm1990/afl-pt) + => This needs Ubuntu 14.04.05 without any updates and the 4.4 kernel. + +2. [https://github.com/hunter-ht-2018/ptfuzzer](https://github.com/hunter-ht-2018/ptfuzzer) + => This needs a 4.14 or 4.15 kernel. The "nopti" kernel boot option must be + used. This one is faster than the other. + +Note that there is also honggfuzz: +[https://github.com/google/honggfuzz](https://github.com/google/honggfuzz). But +its IPT performance is just 6%! + +## Non-AFL++ solutions + +There are many binary-only fuzzing frameworks. Some are great for CTFs but don't +work with large binaries, others are very slow but have good path discovery, +some are very hard to set-up... + + +* Jackalope: + [https://github.com/googleprojectzero/Jackalope](https://github.com/googleprojectzero/Jackalope) +* Manticore: + [https://github.com/trailofbits/manticore](https://github.com/trailofbits/manticore) +* QSYM: + [https://github.com/sslab-gatech/qsym](https://github.com/sslab-gatech/qsym) +* S2E: [https://github.com/S2E](https://github.com/S2E) +* TinyInst: + [https://github.com/googleprojectzero/TinyInst](https://github.com/googleprojectzero/TinyInst) + (Mac/Windows only) +* ... please send me any missing that are good -### More +## Closing words -A more comprehensive description of these and other options can be found in -[binaryonly_fuzzing.md](binaryonly_fuzzing.md). \ No newline at end of file +That's it! News, corrections, updates? Send an email to vh@thc.org. \ No newline at end of file diff --git a/docs/fuzzing_expert.md b/docs/fuzzing_expert.md deleted file mode 100644 index d0d28582..00000000 --- a/docs/fuzzing_expert.md +++ /dev/null @@ -1,626 +0,0 @@ -# Fuzzing with AFL++ - -The following describes how to fuzz with a target if source code is available. -If you have a binary-only target please skip to [#Instrumenting binary-only apps](#Instrumenting binary-only apps) - -Fuzzing source code is a three-step process. - -1. Compile the target with a special compiler that prepares the target to be - fuzzed efficiently. This step is called "instrumenting a target". -2. Prepare the fuzzing by selecting and optimizing the input corpus for the - target. -3. Perform the fuzzing of the target by randomly mutating input and assessing - if a generated input was processed in a new path in the target binary. - -### 1. Instrumenting that target - -#### a) Selecting the best AFL++ compiler for instrumenting the target - -AFL++ comes with a central compiler `afl-cc` that incorporates various different -kinds of compiler targets and and instrumentation options. -The following evaluation flow will help you to select the best possible. - -It is highly recommended to have the newest llvm version possible installed, -anything below 9 is not recommended. - -``` -+--------------------------------+ -| clang/clang++ 11+ is available | --> use LTO mode (afl-clang-lto/afl-clang-lto++) -+--------------------------------+ see [instrumentation/README.lto.md](instrumentation/README.lto.md) - | - | if not, or if the target fails with LTO afl-clang-lto/++ - | - v -+---------------------------------+ -| clang/clang++ 3.8+ is available | --> use LLVM mode (afl-clang-fast/afl-clang-fast++) -+---------------------------------+ see [instrumentation/README.llvm.md](instrumentation/README.llvm.md) - | - | if not, or if the target fails with LLVM afl-clang-fast/++ - | - v - +--------------------------------+ - | gcc 5+ is available | -> use GCC_PLUGIN mode (afl-gcc-fast/afl-g++-fast) - +--------------------------------+ see [instrumentation/README.gcc_plugin.md](instrumentation/README.gcc_plugin.md) and - [instrumentation/README.instrument_list.md](instrumentation/README.instrument_list.md) - | - | if not, or if you do not have a gcc with plugin support - | - v - use GCC mode (afl-gcc/afl-g++) (or afl-clang/afl-clang++ for clang) -``` - -Clickable README links for the chosen compiler: - - * [LTO mode - afl-clang-lto](../instrumentation/README.lto.md) - * [LLVM mode - afl-clang-fast](../instrumentation/README.llvm.md) - * [GCC_PLUGIN mode - afl-gcc-fast](../instrumentation/README.gcc_plugin.md) - * GCC/CLANG modes (afl-gcc/afl-clang) have no README as they have no own features - -You can select the mode for the afl-cc compiler by: - 1. use a symlink to afl-cc: afl-gcc, afl-g++, afl-clang, afl-clang++, - afl-clang-fast, afl-clang-fast++, afl-clang-lto, afl-clang-lto++, - afl-gcc-fast, afl-g++-fast (recommended!) - 2. using the environment variable AFL_CC_COMPILER with MODE - 3. passing --afl-MODE command line options to the compiler via CFLAGS/CXXFLAGS/CPPFLAGS - -MODE can be one of: LTO (afl-clang-lto*), LLVM (afl-clang-fast*), GCC_PLUGIN -(afl-g*-fast) or GCC (afl-gcc/afl-g++) or CLANG(afl-clang/afl-clang++). - -Because no AFL specific command-line options are accepted (beside the ---afl-MODE command), the compile-time tools make fairly broad use of environment -variables, which can be listed with `afl-cc -hh` or by reading [env_variables.md](env_variables.md). - -#### b) Selecting instrumentation options - -The following options are available when you instrument with LTO mode (afl-clang-fast/afl-clang-lto): - - * Splitting integer, string, float and switch comparisons so AFL++ can easier - solve these. This is an important option if you do not have a very good - and large input corpus. This technique is called laf-intel or COMPCOV. - To use this set the following environment variable before compiling the - target: `export AFL_LLVM_LAF_ALL=1` - You can read more about this in [instrumentation/README.laf-intel.md](../instrumentation/README.laf-intel.md) - * A different technique (and usually a better one than laf-intel) is to - instrument the target so that any compare values in the target are sent to - AFL++ which then tries to put these values into the fuzzing data at different - locations. This technique is very fast and good - if the target does not - transform input data before comparison. Therefore this technique is called - `input to state` or `redqueen`. - If you want to use this technique, then you have to compile the target - twice, once specifically with/for this mode by setting `AFL_LLVM_CMPLOG=1`, - and pass this binary to afl-fuzz via the `-c` parameter. - Note that you can compile also just a cmplog binary and use that for both - however there will be a performance penality. - You can read more about this in [instrumentation/README.cmplog.md](../instrumentation/README.cmplog.md) - -If you use LTO, LLVM or GCC_PLUGIN mode (afl-clang-fast/afl-clang-lto/afl-gcc-fast) -you have the option to selectively only instrument parts of the target that you -are interested in: - - * To instrument only those parts of the target that you are interested in - create a file with all the filenames of the source code that should be - instrumented. - For afl-clang-lto and afl-gcc-fast - or afl-clang-fast if a mode other than - DEFAULT/PCGUARD is used or you have llvm > 10.0.0 - just put one - filename or function per line (no directory information necessary for - filenames9, and either set `export AFL_LLVM_ALLOWLIST=allowlist.txt` **or** - `export AFL_LLVM_DENYLIST=denylist.txt` - depending on if you want per - default to instrument unless noted (DENYLIST) or not perform instrumentation - unless requested (ALLOWLIST). - **NOTE:** During optimization functions might be inlined and then would not match! - See [instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md) - -There are many more options and modes available however these are most of the -time less effective. See: - * [instrumentation/README.llvm.md#6) AFL++ Context Sensitive Branch Coverage](../instrumentation/README.llvm.md#6-afl-context-sensitive-branch-coverage). - * [instrumentation/README.llvm.md#7) AFL++ N-Gram Branch Coverage](../instrumentation/README.llvm.md#7-afl-n-gram-branch-coverage) - -#### c) Sanitizers - -It is possible to use sanitizers when instrumenting targets for fuzzing, -which allows you to find bugs that would not necessarily result in a crash. - -Note that sanitizers have a huge impact on CPU (= less executions per second) -and RAM usage. Also you should only run one afl-fuzz instance per sanitizer type. -This is enough because a use-after-free bug will be picked up, e.g. by -ASAN (address sanitizer) anyway when syncing to other fuzzing instances, -so not all fuzzing instances need to be instrumented with ASAN. - -The following sanitizers have built-in support in AFL++: - * ASAN = Address SANitizer, finds memory corruption vulnerabilities like - use-after-free, NULL pointer dereference, buffer overruns, etc. - Enabled with `export AFL_USE_ASAN=1` before compiling. - * MSAN = Memory SANitizer, finds read access to uninitialized memory, eg. - a local variable that is defined and read before it is even set. - Enabled with `export AFL_USE_MSAN=1` before compiling. - * UBSAN = Undefined Behaviour SANitizer, finds instances where - by the - C and C++ standards - undefined behaviour happens, e.g. adding two - signed integers together where the result is larger than a signed integer - can hold. - Enabled with `export AFL_USE_UBSAN=1` before compiling. - * CFISAN = Control Flow Integrity SANitizer, finds instances where the - control flow is found to be illegal. Originally this was rather to - prevent return oriented programming exploit chains from functioning, - in fuzzing this is mostly reduced to detecting type confusion - vulnerabilities - which is however one of the most important and dangerous - C++ memory corruption classes! - Enabled with `export AFL_USE_CFISAN=1` before compiling. - * TSAN = Thread SANitizer, finds thread race conditions. - Enabled with `export AFL_USE_TSAN=1` before compiling. - * LSAN = Leak SANitizer, finds memory leaks in a program. This is not really - a security issue, but for developers this can be very valuable. - Note that unlike the other sanitizers above this needs - `__AFL_LEAK_CHECK();` added to all areas of the target source code where you - find a leak check necessary! - Enabled with `export AFL_USE_LSAN=1` before compiling. - -It is possible to further modify the behaviour of the sanitizers at run-time -by setting `ASAN_OPTIONS=...`, `LSAN_OPTIONS` etc. - the available parameters -can be looked up in the sanitizer documentation of llvm/clang. -afl-fuzz however requires some specific parameters important for fuzzing to be -set. If you want to set your own, it might bail and report what it is missing. - -Note that some sanitizers cannot be used together, e.g. ASAN and MSAN, and -others often cannot work together because of target weirdness, e.g. ASAN and -CFISAN. You might need to experiment which sanitizers you can combine in a -target (which means more instances can be run without a sanitized target, -which is more effective). - -#### d) Modify the target - -If the target has features that make fuzzing more difficult, e.g. -checksums, HMAC, etc. then modify the source code so that checks for these -values are removed. -This can even be done safely for source code used in operational products -by eliminating these checks within these AFL specific blocks: - -``` -#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION - // say that the checksum or HMAC was fine - or whatever is required - // to eliminate the need for the fuzzer to guess the right checksum - return 0; -#endif -``` - -All AFL++ compilers will set this preprocessor definition automatically. - -#### e) Instrument the target - -In this step the target source code is compiled so that it can be fuzzed. - -Basically you have to tell the target build system that the selected AFL++ -compiler is used. Also - if possible - you should always configure the -build system such that the target is compiled statically and not dynamically. -How to do this is described below. - -The #1 rule when instrumenting a target is: avoid instrumenting shared -libraries at all cost. You would need to set LD_LIBRARY_PATH to point to -these, you could accidently type "make install" and install them system wide - -so don't. Really don't. -**Always compile libraries you want to have instrumented as static and link -these to the target program!** - -Then build the target. (Usually with `make`) - -**NOTES** - -1. sometimes configure and build systems are fickle and do not like - stderr output (and think this means a test failure) - which is something - AFL++ likes to do to show statistics. It is recommended to disable AFL++ - instrumentation reporting via `export AFL_QUIET=1`. - -2. sometimes configure and build systems error on warnings - these should be - disabled (e.g. `--disable-werror` for some configure scripts). - -3. in case the configure/build system complains about AFL++'s compiler and - aborts then set `export AFL_NOOPT=1` which will then just behave like the - real compiler. This option has to be unset again before building the target! - -##### configure - -For `configure` build systems this is usually done by: -`CC=afl-clang-fast CXX=afl-clang-fast++ ./configure --disable-shared` - -Note that if you are using the (better) afl-clang-lto compiler you also have to -set AR to llvm-ar[-VERSION] and RANLIB to llvm-ranlib[-VERSION] - as is -described in [instrumentation/README.lto.md](../instrumentation/README.lto.md). - -##### cmake - -For `cmake` build systems this is usually done by: -`mkdir build; cd build; cmake -DCMAKE_C_COMPILER=afl-cc -DCMAKE_CXX_COMPILER=afl-c++ ..` - -Note that if you are using the (better) afl-clang-lto compiler you also have to -set AR to llvm-ar[-VERSION] and RANLIB to llvm-ranlib[-VERSION] - as is -described in [instrumentation/README.lto.md](../instrumentation/README.lto.md). - -##### meson - -For meson you have to set the AFL++ compiler with the very first command! -`CC=afl-cc CXX=afl-c++ meson` - -##### other build systems or if configure/cmake didn't work - -Sometimes cmake and configure do not pick up the AFL++ compiler, or the -ranlib/ar that is needed - because this was just not foreseen by the developer -of the target. Or they have non-standard options. Figure out if there is a -non-standard way to set this, otherwise set up the build normally and edit the -generated build environment afterwards manually to point it to the right compiler -(and/or ranlib and ar). - -#### f) Better instrumentation - -If you just fuzz a target program as-is you are wasting a great opportunity for -much more fuzzing speed. - -This variant requires the usage of afl-clang-lto, afl-clang-fast or afl-gcc-fast. - -It is the so-called `persistent mode`, which is much, much faster but -requires that you code a source file that is specifically calling the target -functions that you want to fuzz, plus a few specific AFL++ functions around -it. See [instrumentation/README.persistent_mode.md](../instrumentation/README.persistent_mode.md) for details. - -Basically if you do not fuzz a target in persistent mode then you are just -doing it for a hobby and not professionally :-). - -#### g) libfuzzer fuzzer harnesses with LLVMFuzzerTestOneInput() - -libfuzzer `LLVMFuzzerTestOneInput()` harnesses are the defacto standard -for fuzzing, and they can be used with AFL++ (and honggfuzz) as well! -Compiling them is as simple as: -``` -afl-clang-fast++ -fsanitize=fuzzer -o harness harness.cpp targetlib.a -``` -You can even use advanced libfuzzer features like `FuzzedDataProvider`, -`LLVMFuzzerMutate()` etc. and they will work! - -The generated binary is fuzzed with afl-fuzz like any other fuzz target. - -Bonus: the target is already optimized for fuzzing due to persistent mode and -shared-memory testcases and hence gives you the fastest speed possible. - -For more information see [utils/aflpp_driver/README.md](../utils/aflpp_driver/README.md) - -### 2. Preparing the fuzzing campaign - -As you fuzz the target with mutated input, having as diverse inputs for the -target as possible improves the efficiency a lot. - -#### a) Collect inputs - -Try to gather valid inputs for the target from wherever you can. E.g. if it is -the PNG picture format try to find as many png files as possible, e.g. from -reported bugs, test suites, random downloads from the internet, unit test -case data - from all kind of PNG software. - -If the input format is not known, you can also modify a target program to write -normal data it receives and processes to a file and use these. - -#### b) Making the input corpus unique - -Use the AFL++ tool `afl-cmin` to remove inputs from the corpus that do not -produce a new path in the target. - -Put all files from step a) into one directory, e.g. INPUTS. - -If the target program is to be called by fuzzing as `bin/target -d INPUTFILE` -the run afl-cmin like this: -`afl-cmin -i INPUTS -o INPUTS_UNIQUE -- bin/target -d @@` -Note that the INPUTFILE argument that the target program would read from has to be set as `@@`. - -If the target reads from stdin instead, just omit the `@@` as this is the -default. - -This step is highly recommended! - -#### c) Minimizing all corpus files - -The shorter the input files that still traverse the same path -within the target, the better the fuzzing will be. This minimization -is done with `afl-tmin` however it is a long process as this has to -be done for every file: - -``` -mkdir input -cd INPUTS_UNIQUE -for i in *; do - afl-tmin -i "$i" -o "../input/$i" -- bin/target -d @@ -done -``` - -This step can also be parallelized, e.g. with `parallel`. -Note that this step is rather optional though. - -#### Done! - -The INPUTS_UNIQUE/ directory from step b) - or even better the directory input/ -if you minimized the corpus in step c) - is the resulting input corpus directory -to be used in fuzzing! :-) - -### 3. Fuzzing the target - -In this final step we fuzz the target. -There are not that many important options to run the target - unless you want -to use many CPU cores/threads for the fuzzing, which will make the fuzzing much -more useful. - -If you just use one CPU for fuzzing, then you are fuzzing just for fun and not -seriously :-) - -#### a) Running afl-fuzz - -Before you do even a test run of afl-fuzz execute `sudo afl-system-config` (on -the host if you execute afl-fuzz in a docker container). This reconfigures the -system for optimal speed - which afl-fuzz checks and bails otherwise. -Set `export AFL_SKIP_CPUFREQ=1` for afl-fuzz to skip this check if you cannot -run afl-system-config with root privileges on the host for whatever reason. - -Note there is also `sudo afl-persistent-config` which sets additional permanent -boot options for a much better fuzzing performance. - -Note that both scripts improve your fuzzing performance but also decrease your -system protection against attacks! So set strong firewall rules and only -expose SSH as a network service if you use these (which is highly recommended). - -If you have an input corpus from step 2 then specify this directory with the `-i` -option. Otherwise create a new directory and create a file with any content -as test data in there. - -If you do not want anything special, the defaults are already usually best, -hence all you need is to specify the seed input directory with the result of -step [2a. Collect inputs](#a-collect-inputs): -`afl-fuzz -i input -o output -- bin/target -d @@` -Note that the directory specified with -o will be created if it does not exist. - -It can be valuable to run afl-fuzz in a screen or tmux shell so you can log off, -or afl-fuzz is not aborted if you are running it in a remote ssh session where -the connection fails in between. -Only do that though once you have verified that your fuzzing setup works! -Simply run it like `screen -dmS afl-main -- afl-fuzz -M main-$HOSTNAME -i ...` -and it will start away in a screen session. To enter this session simply type -`screen -r afl-main`. You see - it makes sense to name the screen session -same as the afl-fuzz -M/-S naming :-) -For more information on screen or tmux please check their documentation. - -If you need to stop and re-start the fuzzing, use the same command line options -(or even change them by selecting a different power schedule or another -mutation mode!) and switch the input directory with a dash (`-`): -`afl-fuzz -i - -o output -- bin/target -d @@` - -Memory limits are not enforced by afl-fuzz by default and the system may run -out of memory. You can decrease the memory with the `-m` option, the value is -in MB. If this is too small for the target, you can usually see this by -afl-fuzz bailing with the message that it could not connect to the forkserver. - -Adding a dictionary is helpful. See the directory [dictionaries/](../dictionaries/) if -something is already included for your data format, and tell afl-fuzz to load -that dictionary by adding `-x dictionaries/FORMAT.dict`. With afl-clang-lto -you have an autodictionary generation for which you need to do nothing except -to use afl-clang-lto as the compiler. You also have the option to generate -a dictionary yourself, see [utils/libtokencap/README.md](../utils/libtokencap/README.md). - -afl-fuzz has a variety of options that help to workaround target quirks like -specific locations for the input file (`-f`), performing deterministic -fuzzing (`-D`) and many more. Check out `afl-fuzz -h`. - -We highly recommend that you set a memory limit for running the target with `-m` -which defines the maximum memory in MB. This prevents a potential -out-of-memory problem for your system plus helps you detect missing `malloc()` -failure handling in the target. -Play around with various -m values until you find one that safely works for all -your input seeds (if you have good ones and then double or quadrouple that. - -By default afl-fuzz never stops fuzzing. To terminate AFL++ simply press Control-C -or send a signal SIGINT. You can limit the number of executions or approximate runtime -in seconds with options also. - -When you start afl-fuzz you will see a user interface that shows what the status -is: - - -All labels are explained in [status_screen.md](status_screen.md). - -#### b) Using multiple cores - -If you want to seriously fuzz then use as many cores/threads as possible to -fuzz your target. - -On the same machine - due to the design of how AFL++ works - there is a maximum -number of CPU cores/threads that are useful, use more and the overall performance -degrades instead. This value depends on the target, and the limit is between 32 -and 64 cores per machine. - -If you have the RAM, it is highly recommended run the instances with a caching -of the testcases. Depending on the average testcase size (and those found -during fuzzing) and their number, a value between 50-500MB is recommended. -You can set the cache size (in MB) by setting the environment variable `AFL_TESTCACHE_SIZE`. - -There should be one main fuzzer (`-M main-$HOSTNAME` option) and as many secondary -fuzzers (eg `-S variant1`) as you have cores that you use. -Every -M/-S entry needs a unique name (that can be whatever), however the same --o output directory location has to be used for all instances. - -For every secondary fuzzer there should be a variation, e.g.: - * one should fuzz the target that was compiled differently: with sanitizers - activated (`export AFL_USE_ASAN=1 ; export AFL_USE_UBSAN=1 ; - export AFL_USE_CFISAN=1`) - * one or two should fuzz the target with CMPLOG/redqueen (see above), at - least one cmplog instance should follow transformations (`-l AT`) - * one to three fuzzers should fuzz a target compiled with laf-intel/COMPCOV - (see above). Important note: If you run more than one laf-intel/COMPCOV - fuzzer and you want them to share their intermediate results, the main - fuzzer (`-M`) must be one of the them! (Although this is not really - recommended.) - -All other secondaries should be used like this: - * A quarter to a third with the MOpt mutator enabled: `-L 0` - * run with a different power schedule, recommended are: - `fast (default), explore, coe, lin, quad, exploit and rare` - which you can set with e.g. `-p explore` - * a few instances should use the old queue cycling with `-Z` - -Also it is recommended to set `export AFL_IMPORT_FIRST=1` to load testcases -from other fuzzers in the campaign first. - -If you have a large corpus, a corpus from a previous run or are fuzzing in -a CI, then also set `export AFL_CMPLOG_ONLY_NEW=1` and `export AFL_FAST_CAL=1`. - -You can also use different fuzzers. -If you are using AFL spinoffs or AFL conforming fuzzers, then just use the -same -o directory and give it a unique `-S` name. -Examples are: - * [Fuzzolic](https://github.com/season-lab/fuzzolic) - * [symcc](https://github.com/eurecom-s3/symcc/) - * [Eclipser](https://github.com/SoftSec-KAIST/Eclipser/) - * [AFLsmart](https://github.com/aflsmart/aflsmart) - * [FairFuzz](https://github.com/carolemieux/afl-rb) - * [Neuzz](https://github.com/Dongdongshe/neuzz) - * [Angora](https://github.com/AngoraFuzzer/Angora) - -A long list can be found at [https://github.com/Microsvuln/Awesome-AFL](https://github.com/Microsvuln/Awesome-AFL) - -However you can also sync AFL++ with honggfuzz, libfuzzer with `-entropic=1`, etc. -Just show the main fuzzer (-M) with the `-F` option where the queue/work -directory of a different fuzzer is, e.g. `-F /src/target/honggfuzz`. -Using honggfuzz (with `-n 1` or `-n 2`) and libfuzzer in parallel is highly -recommended! - -#### c) Using multiple machines for fuzzing - -Maybe you have more than one machine you want to fuzz the same target on. -Simply start the `afl-fuzz` (and perhaps libfuzzer, honggfuzz, ...) -orchestra as you like, just ensure that your have one and only one `-M` -instance per server, and that its name is unique, hence the recommendation -for `-M main-$HOSTNAME`. - -Now there are three strategies on how you can sync between the servers: - * never: sounds weird, but this makes every server an island and has the - chance the each follow different paths into the target. You can make - this even more interesting by even giving different seeds to each server. - * regularly (~4h): this ensures that all fuzzing campaigns on the servers - "see" the same thing. It is like fuzzing on a huge server. - * in intervals of 1/10th of the overall expected runtime of the fuzzing you - sync. This tries a bit to combine both. have some individuality of the - paths each campaign on a server explores, on the other hand if one - gets stuck where another found progress this is handed over making it - unstuck. - -The syncing process itself is very simple. -As the `-M main-$HOSTNAME` instance syncs to all `-S` secondaries as well -as to other fuzzers, you have to copy only this directory to the other -machines. - -Lets say all servers have the `-o out` directory in /target/foo/out, and -you created a file `servers.txt` which contains the hostnames of all -participating servers, plus you have an ssh key deployed to all of them, -then run: -```bash -for FROM in `cat servers.txt`; do - for TO in `cat servers.txt`; do - rsync -rlpogtz --rsh=ssh $FROM:/target/foo/out/main-$FROM $TO:target/foo/out/ - done -done -``` -You can run this manually, per cron job - as you need it. -There is a more complex and configurable script in `utils/distributed_fuzzing`. - -#### d) The status of the fuzz campaign - -AFL++ comes with the `afl-whatsup` script to show the status of the fuzzing -campaign. - -Just supply the directory that afl-fuzz is given with the -o option and -you will see a detailed status of every fuzzer in that campaign plus -a summary. - -To have only the summary use the `-s` switch e.g.: `afl-whatsup -s out/` - -If you have multiple servers then use the command after a sync, or you have -to execute this script per server. - -Another tool to inspect the current state and history of a specific instance -is afl-plot, which generates an index.html file and a graphs that show how -the fuzzing instance is performing. -The syntax is `afl-plot instance_dir web_dir`, e.g. `afl-plot out/default /srv/www/htdocs/plot` - -#### e) Stopping fuzzing, restarting fuzzing, adding new seeds - -To stop an afl-fuzz run, simply press Control-C. - -To restart an afl-fuzz run, just reuse the same command line but replace the -`-i directory` with `-i -` or set `AFL_AUTORESUME=1`. - -If you want to add new seeds to a fuzzing campaign you can run a temporary -fuzzing instance, e.g. when your main fuzzer is using `-o out` and the new -seeds are in `newseeds/` directory: -``` -AFL_BENCH_JUST_ONE=1 AFL_FAST_CAL=1 afl-fuzz -i newseeds -o out -S newseeds -- ./target -``` - -#### f) Checking the coverage of the fuzzing - -The `paths found` value is a bad indicator for checking how good the coverage is. - -A better indicator - if you use default llvm instrumentation with at least -version 9 - is to use `afl-showmap` with the collect coverage option `-C` on -the output directory: -``` -$ afl-showmap -C -i out -o /dev/null -- ./target -params @@ -... -[*] Using SHARED MEMORY FUZZING feature. -[*] Target map size: 9960 -[+] Processed 7849 input files. -[+] Captured 4331 tuples (highest value 255, total values 67130596) in '/dev/nul -l'. -[+] A coverage of 4331 edges were achieved out of 9960 existing (43.48%) with 7849 input files. -``` -It is even better to check out the exact lines of code that have been reached - -and which have not been found so far. - -An "easy" helper script for this is [https://github.com/vanhauser-thc/afl-cov](https://github.com/vanhauser-thc/afl-cov), -just follow the README of that separate project. - -If you see that an important area or a feature has not been covered so far then -try to find an input that is able to reach that and start a new secondary in -that fuzzing campaign with that seed as input, let it run for a few minutes, -then terminate it. The main node will pick it up and make it available to the -other secondary nodes over time. Set `export AFL_NO_AFFINITY=1` or -`export AFL_TRY_AFFINITY=1` if you have no free core. - -Note that in nearly all cases you can never reach full coverage. A lot of -functionality is usually dependent on exclusive options that would need individual -fuzzing campaigns each with one of these options set. E.g. if you fuzz a library to -convert image formats and your target is the png to tiff API then you will not -touch any of the other library APIs and features. - -#### g) How long to fuzz a target? - -This is a difficult question. -Basically if no new path is found for a long time (e.g. for a day or a week) -then you can expect that your fuzzing won't be fruitful anymore. -However often this just means that you should switch out secondaries for -others, e.g. custom mutator modules, sync to very different fuzzers, etc. - -Keep the queue/ directory (for future fuzzings of the same or similar targets) -and use them to seed other good fuzzers like libfuzzer with the -entropic -switch or honggfuzz. - -#### h) Improve the speed! - - * Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 speed increase) - * If you do not use shmem persistent mode, use `AFL_TMPDIR` to point the input file on a tempfs location, see [env_variables.md](env_variables.md) - * Linux: Improve kernel performance: modify `/etc/default/grub`, set `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then `update-grub` and `reboot` (warning: makes the system more insecure) - you can also just run `sudo afl-persistent-config` - * Linux: Running on an `ext2` filesystem with `noatime` mount option will be a bit faster than on any other journaling filesystem - * Use your cores! [b) Using multiple cores](#b-using-multiple-cores) - * Run `sudo afl-system-config` before starting the first afl-fuzz instance after a reboot - -### The End - -Check out the [FAQ](FAQ.md) if it maybe answers your question (that -you might not even have known you had ;-) ). - -This is basically all you need to know to professionally run fuzzing campaigns. -If you want to know more, the tons of texts in [docs/](./) will have you covered. - -Note that there are also a lot of tools out there that help fuzzing with AFL++ -(some might be deprecated or unsupported), see [tools.md](tools.md). \ No newline at end of file diff --git a/docs/fuzzing_in_depth.md b/docs/fuzzing_in_depth.md new file mode 100644 index 00000000..251bbc1d --- /dev/null +++ b/docs/fuzzing_in_depth.md @@ -0,0 +1,853 @@ +# Fuzzing with AFL++ + +The following describes how to fuzz with a target if source code is available. +If you have a binary-only target, please go to +[fuzzing_binary-only_targets.md](fuzzing_binary-only_targets.md). + +Fuzzing source code is a three-step process: + +1. Compile the target with a special compiler that prepares the target to be + fuzzed efficiently. This step is called "instrumenting a target". +2. Prepare the fuzzing by selecting and optimizing the input corpus for the + target. +3. Perform the fuzzing of the target by randomly mutating input and assessing if + a generated input was processed in a new path in the target binary. + +## 0. Common sense risks + +Please keep in mind that, similarly to many other computationally-intensive +tasks, fuzzing may put a strain on your hardware and on the OS. In particular: + +- Your CPU will run hot and will need adequate cooling. In most cases, if + cooling is insufficient or stops working properly, CPU speeds will be + automatically throttled. That said, especially when fuzzing on less suitable + hardware (laptops, smartphones, etc.), it's not entirely impossible for + something to blow up. + +- Targeted programs may end up erratically grabbing gigabytes of memory or + filling up disk space with junk files. AFL++ tries to enforce basic memory + limits, but can't prevent each and every possible mishap. The bottom line is + that you shouldn't be fuzzing on systems where the prospect of data loss is + not an acceptable risk. + +- Fuzzing involves billions of reads and writes to the filesystem. On modern + systems, this will be usually heavily cached, resulting in fairly modest + "physical" I/O - but there are many factors that may alter this equation. It + is your responsibility to monitor for potential trouble; with very heavy I/O, + the lifespan of many HDDs and SSDs may be reduced. + + A good way to monitor disk I/O on Linux is the `iostat` command: + + ```shell + $ iostat -d 3 -x -k [...optional disk ID...] + ``` + + Using the `AFL_TMPDIR` environment variable and a RAM-disk, you can have the + heavy writing done in RAM to prevent the aforementioned wear and tear. For + example, the following line will run a Docker container with all this preset: + + ```shell + # docker run -ti --mount type=tmpfs,destination=/ramdisk -e AFL_TMPDIR=/ramdisk aflplusplus/aflplusplus + ``` + +## 1. Instrumenting the target + +### a) Selecting the best AFL++ compiler for instrumenting the target + +AFL++ comes with a central compiler `afl-cc` that incorporates various different +kinds of compiler targets and and instrumentation options. The following +evaluation flow will help you to select the best possible. + +It is highly recommended to have the newest llvm version possible installed, +anything below 9 is not recommended. + +``` ++--------------------------------+ +| clang/clang++ 11+ is available | --> use LTO mode (afl-clang-lto/afl-clang-lto++) ++--------------------------------+ see [instrumentation/README.lto.md](instrumentation/README.lto.md) + | + | if not, or if the target fails with LTO afl-clang-lto/++ + | + v ++---------------------------------+ +| clang/clang++ 3.8+ is available | --> use LLVM mode (afl-clang-fast/afl-clang-fast++) ++---------------------------------+ see [instrumentation/README.llvm.md](instrumentation/README.llvm.md) + | + | if not, or if the target fails with LLVM afl-clang-fast/++ + | + v + +--------------------------------+ + | gcc 5+ is available | -> use GCC_PLUGIN mode (afl-gcc-fast/afl-g++-fast) + +--------------------------------+ see [instrumentation/README.gcc_plugin.md](instrumentation/README.gcc_plugin.md) and + [instrumentation/README.instrument_list.md](instrumentation/README.instrument_list.md) + | + | if not, or if you do not have a gcc with plugin support + | + v + use GCC mode (afl-gcc/afl-g++) (or afl-clang/afl-clang++ for clang) +``` + +Clickable README links for the chosen compiler: + +* [LTO mode - afl-clang-lto](../instrumentation/README.lto.md) +* [LLVM mode - afl-clang-fast](../instrumentation/README.llvm.md) +* [GCC_PLUGIN mode - afl-gcc-fast](../instrumentation/README.gcc_plugin.md) +* GCC/CLANG modes (afl-gcc/afl-clang) have no README as they have no own + features + +You can select the mode for the afl-cc compiler by: +1. use a symlink to afl-cc: afl-gcc, afl-g++, afl-clang, afl-clang++, + afl-clang-fast, afl-clang-fast++, afl-clang-lto, afl-clang-lto++, + afl-gcc-fast, afl-g++-fast (recommended!) +2. using the environment variable AFL_CC_COMPILER with MODE +3. passing --afl-MODE command line options to the compiler via + CFLAGS/CXXFLAGS/CPPFLAGS + +MODE can be one of: LTO (afl-clang-lto*), LLVM (afl-clang-fast*), GCC_PLUGIN +(afl-g*-fast) or GCC (afl-gcc/afl-g++) or CLANG(afl-clang/afl-clang++). + +Because no AFL specific command-line options are accepted (beside the --afl-MODE +command), the compile-time tools make fairly broad use of environment variables, +which can be listed with `afl-cc -hh` or by reading +[env_variables.md](env_variables.md). + +### b) Selecting instrumentation options + +The following options are available when you instrument with LTO mode +(afl-clang-fast/afl-clang-lto): + +* Splitting integer, string, float and switch comparisons so AFL++ can easier + solve these. This is an important option if you do not have a very good and + large input corpus. This technique is called laf-intel or COMPCOV. To use this + set the following environment variable before compiling the target: `export + AFL_LLVM_LAF_ALL=1` You can read more about this in + [instrumentation/README.laf-intel.md](../instrumentation/README.laf-intel.md). +* A different technique (and usually a better one than laf-intel) is to + instrument the target so that any compare values in the target are sent to + AFL++ which then tries to put these values into the fuzzing data at different + locations. This technique is very fast and good - if the target does not + transform input data before comparison. Therefore this technique is called + `input to state` or `redqueen`. If you want to use this technique, then you + have to compile the target twice, once specifically with/for this mode by + setting `AFL_LLVM_CMPLOG=1`, and pass this binary to afl-fuzz via the `-c` + parameter. Note that you can compile also just a cmplog binary and use that + for both however there will be a performance penality. You can read more about + this in + [instrumentation/README.cmplog.md](../instrumentation/README.cmplog.md). + +If you use LTO, LLVM or GCC_PLUGIN mode +(afl-clang-fast/afl-clang-lto/afl-gcc-fast) you have the option to selectively +only instrument parts of the target that you are interested in: + +* To instrument only those parts of the target that you are interested in create + a file with all the filenames of the source code that should be instrumented. + For afl-clang-lto and afl-gcc-fast - or afl-clang-fast if a mode other than + DEFAULT/PCGUARD is used or you have llvm > 10.0.0 - just put one filename or + function per line (no directory information necessary for filenames9, and + either set `export AFL_LLVM_ALLOWLIST=allowlist.txt` **or** `export + AFL_LLVM_DENYLIST=denylist.txt` - depending on if you want per default to + instrument unless noted (DENYLIST) or not perform instrumentation unless + requested (ALLOWLIST). **NOTE:** During optimization functions might be + inlined and then would not match! See + [instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md) + +There are many more options and modes available however these are most of the +time less effective. See: +* [instrumentation/README.ctx.md](../instrumentation/README.ctx.md) +* [instrumentation/README.ngram.md](../instrumentation/README.ngram.md) + +AFL++ performs "never zero" counting in its bitmap. You can read more about this +here: +* [instrumentation/README.neverzero.md](../instrumentation/README.neverzero.md) + +### c) Selecting sanitizers + +It is possible to use sanitizers when instrumenting targets for fuzzing, which +allows you to find bugs that would not necessarily result in a crash. + +Note that sanitizers have a huge impact on CPU (= less executions per second) +and RAM usage. Also you should only run one afl-fuzz instance per sanitizer +type. This is enough because a use-after-free bug will be picked up, e.g. by +ASAN (address sanitizer) anyway when syncing to other fuzzing instances, so not +all fuzzing instances need to be instrumented with ASAN. + +The following sanitizers have built-in support in AFL++: +* ASAN = Address SANitizer, finds memory corruption vulnerabilities like + use-after-free, NULL pointer dereference, buffer overruns, etc. Enabled with + `export AFL_USE_ASAN=1` before compiling. +* MSAN = Memory SANitizer, finds read access to uninitialized memory, eg. a + local variable that is defined and read before it is even set. Enabled with + `export AFL_USE_MSAN=1` before compiling. +* UBSAN = Undefined Behaviour SANitizer, finds instances where - by the C and + C++ standards - undefined behaviour happens, e.g. adding two signed integers + together where the result is larger than a signed integer can hold. Enabled + with `export AFL_USE_UBSAN=1` before compiling. +* CFISAN = Control Flow Integrity SANitizer, finds instances where the control + flow is found to be illegal. Originally this was rather to prevent return + oriented programming exploit chains from functioning, in fuzzing this is + mostly reduced to detecting type confusion vulnerabilities - which is, + however, one of the most important and dangerous C++ memory corruption + classes! Enabled with `export AFL_USE_CFISAN=1` before compiling. +* TSAN = Thread SANitizer, finds thread race conditions. Enabled with `export + AFL_USE_TSAN=1` before compiling. +* LSAN = Leak SANitizer, finds memory leaks in a program. This is not really a + security issue, but for developers this can be very valuable. Note that unlike + the other sanitizers above this needs `__AFL_LEAK_CHECK();` added to all areas + of the target source code where you find a leak check necessary! Enabled with + `export AFL_USE_LSAN=1` before compiling. + +It is possible to further modify the behaviour of the sanitizers at run-time by +setting `ASAN_OPTIONS=...`, `LSAN_OPTIONS` etc. - the available parameters can +be looked up in the sanitizer documentation of llvm/clang. afl-fuzz, however, +requires some specific parameters important for fuzzing to be set. If you want +to set your own, it might bail and report what it is missing. + +Note that some sanitizers cannot be used together, e.g. ASAN and MSAN, and +others often cannot work together because of target weirdness, e.g. ASAN and +CFISAN. You might need to experiment which sanitizers you can combine in a +target (which means more instances can be run without a sanitized target, which +is more effective). + +### d) Modifying the target + +If the target has features that make fuzzing more difficult, e.g. checksums, +HMAC, etc. then modify the source code so that checks for these values are +removed. This can even be done safely for source code used in operational +products by eliminating these checks within these AFL specific blocks: + +``` +#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION + // say that the checksum or HMAC was fine - or whatever is required + // to eliminate the need for the fuzzer to guess the right checksum + return 0; +#endif +``` + +All AFL++ compilers will set this preprocessor definition automatically. + +### e) Instrumenting the target + +In this step the target source code is compiled so that it can be fuzzed. + +Basically you have to tell the target build system that the selected AFL++ +compiler is used. Also - if possible - you should always configure the build +system such that the target is compiled statically and not dynamically. How to +do this is described below. + +The #1 rule when instrumenting a target is: avoid instrumenting shared libraries +at all cost. You would need to set LD_LIBRARY_PATH to point to these, you could +accidentally type "make install" and install them system wide - so don't. Really +don't. **Always compile libraries you want to have instrumented as static and +link these to the target program!** + +Then build the target. (Usually with `make`) + +**NOTES** + +1. sometimes configure and build systems are fickle and do not like stderr + output (and think this means a test failure) - which is something AFL++ likes + to do to show statistics. It is recommended to disable AFL++ instrumentation + reporting via `export AFL_QUIET=1`. + +2. sometimes configure and build systems error on warnings - these should be + disabled (e.g. `--disable-werror` for some configure scripts). + +3. in case the configure/build system complains about AFL++'s compiler and + aborts then set `export AFL_NOOPT=1` which will then just behave like the + real compiler. This option has to be unset again before building the target! + +#### configure + +For `configure` build systems this is usually done by: +`CC=afl-clang-fast CXX=afl-clang-fast++ ./configure --disable-shared` + +Note that if you are using the (better) afl-clang-lto compiler you also have to +set AR to llvm-ar[-VERSION] and RANLIB to llvm-ranlib[-VERSION] - as is +described in [instrumentation/README.lto.md](../instrumentation/README.lto.md). + +#### cmake + +For `cmake` build systems this is usually done by: +`mkdir build; cd build; cmake -DCMAKE_C_COMPILER=afl-cc -DCMAKE_CXX_COMPILER=afl-c++ ..` + +Note that if you are using the (better) afl-clang-lto compiler you also have to +set AR to llvm-ar[-VERSION] and RANLIB to llvm-ranlib[-VERSION] - as is +described in [instrumentation/README.lto.md](../instrumentation/README.lto.md). + +#### meson + +For meson you have to set the AFL++ compiler with the very first command! +`CC=afl-cc CXX=afl-c++ meson` + +#### other build systems or if configure/cmake didn't work + +Sometimes cmake and configure do not pick up the AFL++ compiler, or the +ranlib/ar that is needed - because this was just not foreseen by the developer +of the target. Or they have non-standard options. Figure out if there is a +non-standard way to set this, otherwise set up the build normally and edit the +generated build environment afterwards manually to point it to the right +compiler (and/or ranlib and ar). + +### f) Better instrumentation + +If you just fuzz a target program as-is you are wasting a great opportunity for +much more fuzzing speed. + +This variant requires the usage of afl-clang-lto, afl-clang-fast or +afl-gcc-fast. + +It is the so-called `persistent mode`, which is much, much faster but requires +that you code a source file that is specifically calling the target functions +that you want to fuzz, plus a few specific AFL++ functions around it. See +[instrumentation/README.persistent_mode.md](../instrumentation/README.persistent_mode.md) +for details. + +Basically if you do not fuzz a target in persistent mode then you are just doing +it for a hobby and not professionally :-). + +### g) libfuzzer fuzzer harnesses with LLVMFuzzerTestOneInput() + +libfuzzer `LLVMFuzzerTestOneInput()` harnesses are the defacto standard +for fuzzing, and they can be used with AFL++ (and honggfuzz) as well! + +Compiling them is as simple as: + +``` +afl-clang-fast++ -fsanitize=fuzzer -o harness harness.cpp targetlib.a +``` + +You can even use advanced libfuzzer features like `FuzzedDataProvider`, +`LLVMFuzzerMutate()` etc. and they will work! + +The generated binary is fuzzed with afl-fuzz like any other fuzz target. + +Bonus: the target is already optimized for fuzzing due to persistent mode and +shared-memory test cases and hence gives you the fastest speed possible. + +For more information, see +[utils/aflpp_driver/README.md](../utils/aflpp_driver/README.md). + +## 2. Preparing the fuzzing campaign + +As you fuzz the target with mutated input, having as diverse inputs for the +target as possible improves the efficiency a lot. + +### a) Collecting inputs + +To operate correctly, the fuzzer requires one or more starting files that +contain a good example of the input data normally expected by the targeted +application. + +Try to gather valid inputs for the target from wherever you can. E.g., if it is +the PNG picture format, try to find as many PNG files as possible, e.g., from +reported bugs, test suites, random downloads from the internet, unit test case +data - from all kind of PNG software. + +If the input format is not known, you can also modify a target program to write +normal data it receives and processes to a file and use these. + +You can find many good examples of starting files in the +[testcases/](../testcases) subdirectory that comes with this tool. + +### b) Making the input corpus unique + +Use the AFL++ tool `afl-cmin` to remove inputs from the corpus that do not +produce a new path in the target. + +Put all files from step a) into one directory, e.g. INPUTS. + +If the target program is to be called by fuzzing as `bin/target -d INPUTFILE` +the run afl-cmin like this: +`afl-cmin -i INPUTS -o INPUTS_UNIQUE -- bin/target -d @@` +Note that the INPUTFILE argument that the target program would read from has to be set as `@@`. + +If the target reads from stdin instead, just omit the `@@` as this is the +default. + +This step is highly recommended! + +### c) Minimizing all corpus files + +The shorter the input files that still traverse the same path within the target, +the better the fuzzing will be. This minimization is done with `afl-tmin` +however it is a long process as this has to be done for every file: + +``` +mkdir input +cd INPUTS_UNIQUE +for i in *; do + afl-tmin -i "$i" -o "../input/$i" -- bin/target -d @@ +done +``` + +This step can also be parallelized, e.g. with `parallel`. Note that this step is +rather optional though. + +### Done! + +The INPUTS_UNIQUE/ directory from step b) - or even better the directory input/ +if you minimized the corpus in step c) - is the resulting input corpus directory +to be used in fuzzing! :-) + +## 3. Fuzzing the target + +In this final step we fuzz the target. There are not that many important options +to run the target - unless you want to use many CPU cores/threads for the +fuzzing, which will make the fuzzing much more useful. + +If you just use one CPU for fuzzing, then you are fuzzing just for fun and not +seriously :-) + +### a) Running afl-fuzz + +Before you do even a test run of afl-fuzz execute `sudo afl-system-config` (on +the host if you execute afl-fuzz in a docker container). This reconfigures the +system for optimal speed - which afl-fuzz checks and bails otherwise. Set +`export AFL_SKIP_CPUFREQ=1` for afl-fuzz to skip this check if you cannot run +afl-system-config with root privileges on the host for whatever reason. + +Note there is also `sudo afl-persistent-config` which sets additional permanent +boot options for a much better fuzzing performance. + +Note that both scripts improve your fuzzing performance but also decrease your +system protection against attacks! So set strong firewall rules and only expose +SSH as a network service if you use these (which is highly recommended). + +If you have an input corpus from step 2 then specify this directory with the +`-i` option. Otherwise create a new directory and create a file with any content +as test data in there. + +If you do not want anything special, the defaults are already usually best, +hence all you need is to specify the seed input directory with the result of +step [2a) Collect inputs](#a-collect-inputs): +`afl-fuzz -i input -o output -- bin/target -d @@` +Note that the directory specified with -o will be created if it does not exist. + +It can be valuable to run afl-fuzz in a screen or tmux shell so you can log off, +or afl-fuzz is not aborted if you are running it in a remote ssh session where +the connection fails in between. +Only do that though once you have verified that your fuzzing setup works! +Simply run it like `screen -dmS afl-main -- afl-fuzz -M main-$HOSTNAME -i ...` +and it will start away in a screen session. To enter this session simply type +`screen -r afl-main`. You see - it makes sense to name the screen session +same as the afl-fuzz -M/-S naming :-) +For more information on screen or tmux please check their documentation. + +If you need to stop and re-start the fuzzing, use the same command line options +(or even change them by selecting a different power schedule or another mutation +mode!) and switch the input directory with a dash (`-`): +`afl-fuzz -i - -o output -- bin/target -d @@` + +Adding a dictionary is helpful. See the directory +[dictionaries/](../dictionaries/) if something is already included for your data +format, and tell afl-fuzz to load that dictionary by adding `-x +dictionaries/FORMAT.dict`. With afl-clang-lto you have an autodictionary +generation for which you need to do nothing except to use afl-clang-lto as the +compiler. You also have the option to generate a dictionary yourself, see +[utils/libtokencap/README.md](../utils/libtokencap/README.md). + +afl-fuzz has a variety of options that help to workaround target quirks like +specific locations for the input file (`-f`), performing deterministic fuzzing +(`-D`) and many more. Check out `afl-fuzz -h`. + +We highly recommend that you set a memory limit for running the target with `-m` +which defines the maximum memory in MB. This prevents a potential out-of-memory +problem for your system plus helps you detect missing `malloc()` failure +handling in the target. Play around with various -m values until you find one +that safely works for all your input seeds (if you have good ones and then +double or quadruple that. + +By default afl-fuzz never stops fuzzing. To terminate AFL++ simply press +Control-C or send a signal SIGINT. You can limit the number of executions or +approximate runtime in seconds with options also. + +When you start afl-fuzz you will see a user interface that shows what the status +is: + + +All labels are explained in [status_screen.md](status_screen.md). + +### b) Keeping memory use and timeouts in check + +Memory limits are not enforced by afl-fuzz by default and the system may run out +of memory. You can decrease the memory with the `-m` option, the value is in MB. +If this is too small for the target, you can usually see this by afl-fuzz +bailing with the message that it could not connect to the forkserver. + +Consider setting low values for `-m` and `-t`. + +For programs that are nominally very fast, but get sluggish for some inputs, you +can also try setting `-t` values that are more punishing than what `afl-fuzz` +dares to use on its own. On fast and idle machines, going down to `-t 5` may be +a viable plan. + +The `-m` parameter is worth looking at, too. Some programs can end up spending a +fair amount of time allocating and initializing megabytes of memory when +presented with pathological inputs. Low `-m` values can make them give up sooner +and not waste CPU time. + +### c) Using multiple cores + +If you want to seriously fuzz then use as many cores/threads as possible to fuzz +your target. + +On the same machine - due to the design of how AFL++ works - there is a maximum +number of CPU cores/threads that are useful, use more and the overall +performance degrades instead. This value depends on the target, and the limit is +between 32 and 64 cores per machine. + +If you have the RAM, it is highly recommended run the instances with a caching +of the test cases. Depending on the average test case size (and those found +during fuzzing) and their number, a value between 50-500MB is recommended. You +can set the cache size (in MB) by setting the environment variable +`AFL_TESTCACHE_SIZE`. + +There should be one main fuzzer (`-M main-$HOSTNAME` option) and as many +secondary fuzzers (e.g. `-S variant1`) as you have cores that you use. Every +-M/-S entry needs a unique name (that can be whatever), however, the same -o +output directory location has to be used for all instances. + +For every secondary fuzzer there should be a variation, e.g.: +* one should fuzz the target that was compiled differently: with sanitizers + activated (`export AFL_USE_ASAN=1 ; export AFL_USE_UBSAN=1 ; export + AFL_USE_CFISAN=1`) +* one or two should fuzz the target with CMPLOG/redqueen (see above), at least + one cmplog instance should follow transformations (`-l AT`) +* one to three fuzzers should fuzz a target compiled with laf-intel/COMPCOV (see + above). Important note: If you run more than one laf-intel/COMPCOV fuzzer and + you want them to share their intermediate results, the main fuzzer (`-M`) must + be one of them! (Although this is not really recommended.) + +All other secondaries should be used like this: +* a quarter to a third with the MOpt mutator enabled: `-L 0` +* run with a different power schedule, recommended are: + `fast (default), explore, coe, lin, quad, exploit and rare` which you can set + with e.g. `-p explore` +* a few instances should use the old queue cycling with `-Z` + +Also, it is recommended to set `export AFL_IMPORT_FIRST=1` to load test cases +from other fuzzers in the campaign first. + +If you have a large corpus, a corpus from a previous run or are fuzzing in +a CI, then also set `export AFL_CMPLOG_ONLY_NEW=1` and `export AFL_FAST_CAL=1`. + +You can also use different fuzzers. If you are using AFL spinoffs or AFL +conforming fuzzers, then just use the same -o directory and give it a unique +`-S` name. Examples are: +* [Fuzzolic](https://github.com/season-lab/fuzzolic) +* [symcc](https://github.com/eurecom-s3/symcc/) +* [Eclipser](https://github.com/SoftSec-KAIST/Eclipser/) +* [AFLsmart](https://github.com/aflsmart/aflsmart) +* [FairFuzz](https://github.com/carolemieux/afl-rb) +* [Neuzz](https://github.com/Dongdongshe/neuzz) +* [Angora](https://github.com/AngoraFuzzer/Angora) + +A long list can be found at +[https://github.com/Microsvuln/Awesome-AFL](https://github.com/Microsvuln/Awesome-AFL). + +However, you can also sync AFL++ with honggfuzz, libfuzzer with `-entropic=1`, +etc. Just show the main fuzzer (-M) with the `-F` option where the queue/work +directory of a different fuzzer is, e.g. `-F /src/target/honggfuzz`. Using +honggfuzz (with `-n 1` or `-n 2`) and libfuzzer in parallel is highly +recommended! + +### d) Using multiple machines for fuzzing + +Maybe you have more than one machine you want to fuzz the same target on. +Simply start the `afl-fuzz` (and perhaps libfuzzer, honggfuzz, ...) +orchestra as you like, just ensure that your have one and only one `-M` +instance per server, and that its name is unique, hence the recommendation +for `-M main-$HOSTNAME`. + +Now there are three strategies on how you can sync between the servers: +* never: sounds weird, but this makes every server an island and has the chance + the each follow different paths into the target. You can make this even more + interesting by even giving different seeds to each server. +* regularly (~4h): this ensures that all fuzzing campaigns on the servers "see" + the same thing. It is like fuzzing on a huge server. +* in intervals of 1/10th of the overall expected runtime of the fuzzing you + sync. This tries a bit to combine both. have some individuality of the paths + each campaign on a server explores, on the other hand if one gets stuck where + another found progress this is handed over making it unstuck. + +The syncing process itself is very simple. As the `-M main-$HOSTNAME` instance +syncs to all `-S` secondaries as well as to other fuzzers, you have to copy only +this directory to the other machines. + +Lets say all servers have the `-o out` directory in /target/foo/out, and you +created a file `servers.txt` which contains the hostnames of all participating +servers, plus you have an ssh key deployed to all of them, then run: + +```bash +for FROM in `cat servers.txt`; do + for TO in `cat servers.txt`; do + rsync -rlpogtz --rsh=ssh $FROM:/target/foo/out/main-$FROM $TO:target/foo/out/ + done +done +``` + +You can run this manually, per cron job - as you need it. There is a more +complex and configurable script in `utils/distributed_fuzzing`. + +### e) The status of the fuzz campaign + +AFL++ comes with the `afl-whatsup` script to show the status of the fuzzing +campaign. + +Just supply the directory that afl-fuzz is given with the `-o` option and you +will see a detailed status of every fuzzer in that campaign plus a summary. + +To have only the summary, use the `-s` switch, e.g., `afl-whatsup -s out/`. + +If you have multiple servers, then use the command after a sync or you have to +execute this script per server. + +Another tool to inspect the current state and history of a specific instance is +afl-plot, which generates an index.html file and a graphs that show how the +fuzzing instance is performing. The syntax is `afl-plot instance_dir web_dir`, +e.g., `afl-plot out/default /srv/www/htdocs/plot`. + +### f) Stopping fuzzing, restarting fuzzing, adding new seeds + +To stop an afl-fuzz run, simply press Control-C. + +To restart an afl-fuzz run, just reuse the same command line but replace the `-i +directory` with `-i -` or set `AFL_AUTORESUME=1`. + +If you want to add new seeds to a fuzzing campaign you can run a temporary +fuzzing instance, e.g. when your main fuzzer is using `-o out` and the new seeds +are in `newseeds/` directory: + +``` +AFL_BENCH_JUST_ONE=1 AFL_FAST_CAL=1 afl-fuzz -i newseeds -o out -S newseeds -- ./target +``` + +### g) Checking the coverage of the fuzzing + +The `paths found` value is a bad indicator for checking how good the coverage +is. + +A better indicator - if you use default llvm instrumentation with at least +version 9 - is to use `afl-showmap` with the collect coverage option `-C` on the +output directory: + +``` +$ afl-showmap -C -i out -o /dev/null -- ./target -params @@ +... +[*] Using SHARED MEMORY FUZZING feature. +[*] Target map size: 9960 +[+] Processed 7849 input files. +[+] Captured 4331 tuples (highest value 255, total values 67130596) in '/dev/nul +l'. +[+] A coverage of 4331 edges were achieved out of 9960 existing (43.48%) with 7849 input files. +``` + +It is even better to check out the exact lines of code that have been reached - +and which have not been found so far. + +An "easy" helper script for this is +[https://github.com/vanhauser-thc/afl-cov](https://github.com/vanhauser-thc/afl-cov), +just follow the README of that separate project. + +If you see that an important area or a feature has not been covered so far then +try to find an input that is able to reach that and start a new secondary in +that fuzzing campaign with that seed as input, let it run for a few minutes, +then terminate it. The main node will pick it up and make it available to the +other secondary nodes over time. Set `export AFL_NO_AFFINITY=1` or `export +AFL_TRY_AFFINITY=1` if you have no free core. + +Note that in nearly all cases you can never reach full coverage. A lot of +functionality is usually dependent on exclusive options that would need +individual fuzzing campaigns each with one of these options set. E.g., if you +fuzz a library to convert image formats and your target is the png to tiff API +then you will not touch any of the other library APIs and features. + +### h) How long to fuzz a target? + +This is a difficult question. Basically if no new path is found for a long time +(e.g. for a day or a week) then you can expect that your fuzzing won't be +fruitful anymore. However, often this just means that you should switch out +secondaries for others, e.g. custom mutator modules, sync to very different +fuzzers, etc. + +Keep the queue/ directory (for future fuzzings of the same or similar targets) +and use them to seed other good fuzzers like libfuzzer with the -entropic switch +or honggfuzz. + +### i) Improve the speed! + +* Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 + speed increase) +* If you do not use shmem persistent mode, use `AFL_TMPDIR` to point the input + file on a tempfs location, see [env_variables.md](env_variables.md) +* Linux: Improve kernel performance: modify `/etc/default/grub`, set + `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off + mitigations=off no_stf_barrier noibpb noibrs nopcid nopti + nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off + spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then + `update-grub` and `reboot` (warning: makes the system more insecure) - you can + also just run `sudo afl-persistent-config` +* Linux: Running on an `ext2` filesystem with `noatime` mount option will be a + bit faster than on any other journaling filesystem +* Use your cores! [3c) Using multiple cores](#c-using-multiple-cores) +* Run `sudo afl-system-config` before starting the first afl-fuzz instance after + a reboot + +### j) Going beyond crashes + +Fuzzing is a wonderful and underutilized technique for discovering non-crashing +design and implementation errors, too. Quite a few interesting bugs have been +found by modifying the target programs to call `abort()` when say: + +- Two bignum libraries produce different outputs when given the same + fuzzer-generated input. + +- An image library produces different outputs when asked to decode the same + input image several times in a row. + +- A serialization/deserialization library fails to produce stable outputs when + iteratively serializing and deserializing fuzzer-supplied data. + +- A compression library produces an output inconsistent with the input file when + asked to compress and then decompress a particular blob. + +Implementing these or similar sanity checks usually takes very little time; if +you are the maintainer of a particular package, you can make this code +conditional with `#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` (a flag also +shared with libfuzzer and honggfuzz) or `#ifdef __AFL_COMPILER` (this one is +just for AFL++). + +### k) Known limitations & areas for improvement + +Here are some of the most important caveats for AFL++: + +- AFL++ detects faults by checking for the first spawned process dying due to a + signal (SIGSEGV, SIGABRT, etc). Programs that install custom handlers for + these signals may need to have the relevant code commented out. In the same + vein, faults in child processes spawned by the fuzzed target may evade + detection unless you manually add some code to catch that. + +- As with any other brute-force tool, the fuzzer offers limited coverage if + encryption, checksums, cryptographic signatures, or compression are used to + wholly wrap the actual data format to be tested. + + To work around this, you can comment out the relevant checks (see + utils/libpng_no_checksum/ for inspiration); if this is not possible, you can + also write a postprocessor, one of the hooks of custom mutators. See + [custom_mutators.md](custom_mutators.md) on how to use + `AFL_CUSTOM_MUTATOR_LIBRARY`. + +- There are some unfortunate trade-offs with ASAN and 64-bit binaries. This + isn't due to any specific fault of afl-fuzz. + +- There is no direct support for fuzzing network services, background daemons, + or interactive apps that require UI interaction to work. You may need to make + simple code changes to make them behave in a more traditional way. Preeny may + offer a relatively simple option, too - see: + [https://github.com/zardus/preeny](https://github.com/zardus/preeny) + + Some useful tips for modifying network-based services can be also found at: + [https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop](https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop) + +- Occasionally, sentient machines rise against their creators. If this happens + to you, please consult + [https://lcamtuf.coredump.cx/prep/](https://lcamtuf.coredump.cx/prep/). + +Beyond this, see [INSTALL.md](INSTALL.md) for platform-specific tips. + +## 4. Triaging crashes + +The coverage-based grouping of crashes usually produces a small data set that +can be quickly triaged manually or with a very simple GDB or Valgrind script. +Every crash is also traceable to its parent non-crashing test case in the queue, +making it easier to diagnose faults. + +Having said that, it's important to acknowledge that some fuzzing crashes can be +difficult to quickly evaluate for exploitability without a lot of debugging and +code analysis work. To assist with this task, afl-fuzz supports a very unique +"crash exploration" mode enabled with the -C flag. + +In this mode, the fuzzer takes one or more crashing test cases as the input and +uses its feedback-driven fuzzing strategies to very quickly enumerate all code +paths that can be reached in the program while keeping it in the crashing state. + +Mutations that do not result in a crash are rejected; so are any changes that do +not affect the execution path. + +The output is a small corpus of files that can be very rapidly examined to see +what degree of control the attacker has over the faulting address, or whether it +is possible to get past an initial out-of-bounds read - and see what lies +beneath. + +Oh, one more thing: for test case minimization, give afl-tmin a try. The tool +can be operated in a very simple way: + +```shell +./afl-tmin -i test_case -o minimized_result -- /path/to/program [...] +``` + +The tool works with crashing and non-crashing test cases alike. In the crash +mode, it will happily accept instrumented and non-instrumented binaries. In the +non-crashing mode, the minimizer relies on standard AFL++ instrumentation to +make the file simpler without altering the execution path. + +The minimizer accepts the -m, -t, -f and @@ syntax in a manner compatible with +afl-fuzz. + +Another tool in AFL++ is the afl-analyze tool. It takes an input file, attempts +to sequentially flip bytes, and observes the behavior of the tested program. It +then color-codes the input based on which sections appear to be critical, and +which are not; while not bulletproof, it can often offer quick insights into +complex file formats. + + +## 5. CI fuzzing + +Some notes on CI fuzzing - this fuzzing is different to normal fuzzing campaigns +as these are much shorter runnings. + +1. Always: + * LTO has a much longer compile time which is diametrical to short fuzzing - + hence use afl-clang-fast instead. + * If you compile with CMPLOG, then you can save fuzzing time and reuse that + compiled target for both the `-c` option and the main fuzz target. This + will impact the speed by ~15% though. + * `AFL_FAST_CAL` - Enable fast calibration, this halves the time the + saturated corpus needs to be loaded. + * `AFL_CMPLOG_ONLY_NEW` - only perform cmplog on new found paths, not the + initial corpus as this very likely has been done for them already. + * Keep the generated corpus, use afl-cmin and reuse it every time! + +2. Additionally randomize the AFL++ compilation options, e.g.: + * 40% for `AFL_LLVM_CMPLOG` + * 10% for `AFL_LLVM_LAF_ALL` + +3. Also randomize the afl-fuzz runtime options, e.g.: + * 65% for `AFL_DISABLE_TRIM` + * 50% use a dictionary generated by `AFL_LLVM_DICT2FILE` + * 40% use MOpt (`-L 0`) + * 40% for `AFL_EXPAND_HAVOC_NOW` + * 20% for old queue processing (`-Z`) + * for CMPLOG targets, 60% for `-l 2`, 40% for `-l 3` + +4. Do *not* run any `-M` modes, just running `-S` modes is better for CI + fuzzing. `-M` enables old queue handling etc. which is good for a fuzzing + campaign but not good for short CI runs. + +How this can look like can, e.g., be seen at AFL++'s setup in Google's +[oss-fuzz](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/compile_afl) +and +[clusterfuzz](https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/bot/fuzzers/afl/launcher.py). + +## The End + +Check out the [FAQ](FAQ.md) if it maybe answers your question (that you might +not even have known you had ;-) ). + +This is basically all you need to know to professionally run fuzzing campaigns. +If you want to know more, the tons of texts in [docs/](./) will have you +covered. + +Note that there are also a lot of tools out there that help fuzzing with AFL++ +(some might be deprecated or unsupported), see +[third_party_tools.md](third_party_tools.md). \ No newline at end of file diff --git a/docs/important_changes.md b/docs/important_changes.md index 0c5c2243..877dfab2 100644 --- a/docs/important_changes.md +++ b/docs/important_changes.md @@ -36,7 +36,7 @@ behaviours and defaults: shared libraries, etc. Additionally QEMU 5.1 supports more CPU targets so this is really worth it. * When instrumenting targets, afl-cc will not supersede optimizations anymore - if any were given. This allows to fuzz targets build regularly like those + if any were given. This allows to fuzz targets build regularly like those for debug or release versions. * afl-fuzz: * if neither -M or -S is specified, `-S default` is assumed, so more @@ -47,7 +47,7 @@ behaviours and defaults: * -m none is now default, set memory limits (in MB) with e.g. -m 250 * deterministic fuzzing is now disabled by default (unless using -M) and can be enabled with -D - * a caching of testcases can now be performed and can be modified by + * a caching of test cases can now be performed and can be modified by editing config.h for TESTCASE_CACHE or by specifying the env variable `AFL_TESTCACHE_SIZE` (in MB). Good values are between 50-500 (default: 50). * -M mains do not perform trimming diff --git a/docs/interpreting_output.md b/docs/interpreting_output.md deleted file mode 100644 index 4bd705f2..00000000 --- a/docs/interpreting_output.md +++ /dev/null @@ -1,71 +0,0 @@ -# Interpreting output - -See the [status_screen.md](status_screen.md) file for information on -how to interpret the displayed stats and monitor the health of the process. Be -sure to consult this file especially if any UI elements are highlighted in red. - -The fuzzing process will continue until you press Ctrl-C. At a minimum, you want -to allow the fuzzer to complete one queue cycle, which may take anywhere from a -couple of hours to a week or so. - -There are three subdirectories created within the output directory and updated -in real-time: - - - queue/ - test cases for every distinctive execution path, plus all the - starting files given by the user. This is the synthesized corpus - mentioned in section 2. - - Before using this corpus for any other purposes, you can shrink - it to a smaller size using the afl-cmin tool. The tool will find - a smaller subset of files offering equivalent edge coverage. - - - crashes/ - unique test cases that cause the tested program to receive a - fatal signal (e.g., SIGSEGV, SIGILL, SIGABRT). The entries are - grouped by the received signal. - - - hangs/ - unique test cases that cause the tested program to time out. The - default time limit before something is classified as a hang is - the larger of 1 second and the value of the -t parameter. - The value can be fine-tuned by setting AFL_HANG_TMOUT, but this - is rarely necessary. - -Crashes and hangs are considered "unique" if the associated execution paths -involve any state transitions not seen in previously-recorded faults. If a -single bug can be reached in multiple ways, there will be some count inflation -early in the process, but this should quickly taper off. - -The file names for crashes and hangs are correlated with the parent, non-faulting -queue entries. This should help with debugging. - -When you can't reproduce a crash found by afl-fuzz, the most likely cause is -that you are not setting the same memory limit as used by the tool. Try: - -```shell -LIMIT_MB=50 -( ulimit -Sv $[LIMIT_MB << 10]; /path/to/tested_binary ... ) -``` - -Change LIMIT_MB to match the -m parameter passed to afl-fuzz. On OpenBSD, -also change -Sv to -Sd. - -Any existing output directory can be also used to resume aborted jobs; try: - -```shell -./afl-fuzz -i- -o existing_output_dir [...etc...] -``` - -If you have gnuplot installed, you can also generate some pretty graphs for any -active fuzzing task using afl-plot. For an example of how this looks like, -see [https://lcamtuf.coredump.cx/afl/plot/](https://lcamtuf.coredump.cx/afl/plot/). - -You can also manually build and install afl-plot-ui, which is a helper utility -for showing the graphs generated by afl-plot in a graphical window using GTK. -You can build and install it as follows - -```shell -sudo apt install libgtk-3-0 libgtk-3-dev pkg-config -cd utils/plot_ui -make -cd ../../ -sudo make install -``` diff --git a/docs/known_limitations.md b/docs/known_limitations.md deleted file mode 100644 index a68c0a85..00000000 --- a/docs/known_limitations.md +++ /dev/null @@ -1,36 +0,0 @@ -# Known limitations & areas for improvement - -Here are some of the most important caveats for AFL: - - - AFL++ detects faults by checking for the first spawned process dying due to - a signal (SIGSEGV, SIGABRT, etc). Programs that install custom handlers for - these signals may need to have the relevant code commented out. In the same - vein, faults in child processes spawned by the fuzzed target may evade - detection unless you manually add some code to catch that. - - - As with any other brute-force tool, the fuzzer offers limited coverage if - encryption, checksums, cryptographic signatures, or compression are used to - wholly wrap the actual data format to be tested. - - To work around this, you can comment out the relevant checks (see - utils/libpng_no_checksum/ for inspiration); if this is not possible, - you can also write a postprocessor, one of the hooks of custom mutators. - See [custom_mutators.md](custom_mutators.md) on how to use - `AFL_CUSTOM_MUTATOR_LIBRARY` - - - There are some unfortunate trade-offs with ASAN and 64-bit binaries. This - isn't due to any specific fault of afl-fuzz. - - - There is no direct support for fuzzing network services, background - daemons, or interactive apps that require UI interaction to work. You may - need to make simple code changes to make them behave in a more traditional - way. Preeny may offer a relatively simple option, too - see: - [https://github.com/zardus/preeny](https://github.com/zardus/preeny) - - Some useful tips for modifying network-based services can be also found at: - [https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop](https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop) - - - Occasionally, sentient machines rise against their creators. If this - happens to you, please consult [https://lcamtuf.coredump.cx/prep/](https://lcamtuf.coredump.cx/prep/). - -Beyond this, see [INSTALL.md](INSTALL.md) for platform-specific tips. diff --git a/docs/parallel_fuzzing.md b/docs/parallel_fuzzing.md deleted file mode 100644 index d24f2837..00000000 --- a/docs/parallel_fuzzing.md +++ /dev/null @@ -1,258 +0,0 @@ -# Tips for parallel fuzzing - -This document talks about synchronizing afl-fuzz jobs on a single machine -or across a fleet of systems. See README.md for the general instruction manual. - -Note that this document is rather outdated. please refer to the main document -section on multiple core usage [fuzzing_expert.md#Using multiple cores](fuzzing_expert.md#b-using-multiple-cores) -for up to date strategies! - -## 1) Introduction - -Every copy of afl-fuzz will take up one CPU core. This means that on an -n-core system, you can almost always run around n concurrent fuzzing jobs with -virtually no performance hit (you can use the afl-gotcpu tool to make sure). - -In fact, if you rely on just a single job on a multi-core system, you will -be underutilizing the hardware. So, parallelization is always the right way to -go. - -When targeting multiple unrelated binaries or using the tool in -"non-instrumented" (-n) mode, it is perfectly fine to just start up several -fully separate instances of afl-fuzz. The picture gets more complicated when -you want to have multiple fuzzers hammering a common target: if a hard-to-hit -but interesting test case is synthesized by one fuzzer, the remaining instances -will not be able to use that input to guide their work. - -To help with this problem, afl-fuzz offers a simple way to synchronize test -cases on the fly. - -It is a good idea to use different power schedules if you run several instances -in parallel (`-p` option). - -Alternatively running other AFL spinoffs in parallel can be of value, -e.g. Angora (https://github.com/AngoraFuzzer/Angora/) - -## 2) Single-system parallelization - -If you wish to parallelize a single job across multiple cores on a local -system, simply create a new, empty output directory ("sync dir") that will be -shared by all the instances of afl-fuzz; and then come up with a naming scheme -for every instance - say, "fuzzer01", "fuzzer02", etc. - -Run the first one ("main node", -M) like this: - -``` -./afl-fuzz -i testcase_dir -o sync_dir -M fuzzer01 [...other stuff...] -``` - -...and then, start up secondary (-S) instances like this: - -``` -./afl-fuzz -i testcase_dir -o sync_dir -S fuzzer02 [...other stuff...] -./afl-fuzz -i testcase_dir -o sync_dir -S fuzzer03 [...other stuff...] -``` - -Each fuzzer will keep its state in a separate subdirectory, like so: - - /path/to/sync_dir/fuzzer01/ - -Each instance will also periodically rescan the top-level sync directory -for any test cases found by other fuzzers - and will incorporate them into -its own fuzzing when they are deemed interesting enough. -For performance reasons only -M main node syncs the queue with everyone, the --S secondary nodes will only sync from the main node. - -The difference between the -M and -S modes is that the main instance will -still perform deterministic checks; while the secondary instances will -proceed straight to random tweaks. - -Note that you must always have one -M main instance! -Running multiple -M instances is wasteful! - -You can also monitor the progress of your jobs from the command line with the -provided afl-whatsup tool. When the instances are no longer finding new paths, -it's probably time to stop. - -WARNING: Exercise caution when explicitly specifying the -f option. Each fuzzer -must use a separate temporary file; otherwise, things will go south. One safe -example may be: - -``` -./afl-fuzz [...] -S fuzzer10 -f file10.txt ./fuzzed/binary @@ -./afl-fuzz [...] -S fuzzer11 -f file11.txt ./fuzzed/binary @@ -./afl-fuzz [...] -S fuzzer12 -f file12.txt ./fuzzed/binary @@ -``` - -This is not a concern if you use @@ without -f and let afl-fuzz come up with the -file name. - -## 3) Multiple -M mains - - -There is support for parallelizing the deterministic checks. -This is only needed where - - 1. many new paths are found fast over a long time and it looks unlikely that - main node will ever catch up, and - 2. deterministic fuzzing is actively helping path discovery (you can see this - in the main node for the first for lines in the "fuzzing strategy yields" - section. If the ration `found/attemps` is high, then it is effective. It - most commonly isn't.) - -Only if both are true it is beneficial to have more than one main. -You can leverage this by creating -M instances like so: - -``` -./afl-fuzz -i testcase_dir -o sync_dir -M mainA:1/3 [...] -./afl-fuzz -i testcase_dir -o sync_dir -M mainB:2/3 [...] -./afl-fuzz -i testcase_dir -o sync_dir -M mainC:3/3 [...] -``` - -... where the first value after ':' is the sequential ID of a particular main -instance (starting at 1), and the second value is the total number of fuzzers to -distribute the deterministic fuzzing across. Note that if you boot up fewer -fuzzers than indicated by the second number passed to -M, you may end up with -poor coverage. - -## 4) Syncing with non-AFL fuzzers or independant instances - -A -M main node can be told with the `-F other_fuzzer_queue_directory` option -to sync results from other fuzzers, e.g. libfuzzer or honggfuzz. - -Only the specified directory will by synced into afl, not subdirectories. -The specified directory does not need to exist yet at the start of afl. - -The `-F` option can be passed to the main node several times. - -## 5) Multi-system parallelization - -The basic operating principle for multi-system parallelization is similar to -the mechanism explained in section 2. The key difference is that you need to -write a simple script that performs two actions: - - - Uses SSH with authorized_keys to connect to every machine and retrieve - a tar archive of the /path/to/sync_dir/<main_node(s)> directory local to - the machine. - It is best to use a naming scheme that includes host name and it's being - a main node (e.g. main1, main2) in the fuzzer ID, so that you can do - something like: - - ```sh - for host in `cat HOSTLIST`; do - ssh user@$host "tar -czf - sync/$host_main*/" > $host.tgz - done - ``` - - - Distributes and unpacks these files on all the remaining machines, e.g.: - - ```sh - for srchost in `cat HOSTLIST`; do - for dsthost in `cat HOSTLIST`; do - test "$srchost" = "$dsthost" && continue - ssh user@$srchost 'tar -kxzf -' < $dsthost.tgz - done - done - ``` - -There is an example of such a script in utils/distributed_fuzzing/. - -There are other (older) more featured, experimental tools: - * https://github.com/richo/roving - * https://github.com/MartijnB/disfuzz-afl - -However these do not support syncing just main nodes (yet). - -When developing custom test case sync code, there are several optimizations -to keep in mind: - - - The synchronization does not have to happen very often; running the - task every 60 minutes or even less often at later fuzzing stages is - fine - - - There is no need to synchronize crashes/ or hangs/; you only need to - copy over queue/* (and ideally, also fuzzer_stats). - - - It is not necessary (and not advisable!) to overwrite existing files; - the -k option in tar is a good way to avoid that. - - - There is no need to fetch directories for fuzzers that are not running - locally on a particular machine, and were simply copied over onto that - system during earlier runs. - - - For large fleets, you will want to consolidate tarballs for each host, - as this will let you use n SSH connections for sync, rather than n*(n-1). - - You may also want to implement staged synchronization. For example, you - could have 10 groups of systems, with group 1 pushing test cases only - to group 2; group 2 pushing them only to group 3; and so on, with group - eventually 10 feeding back to group 1. - - This arrangement would allow test interesting cases to propagate across - the fleet without having to copy every fuzzer queue to every single host. - - - You do not want a "main" instance of afl-fuzz on every system; you should - run them all with -S, and just designate a single process somewhere within - the fleet to run with -M. - - - Syncing is only necessary for the main nodes on a system. It is possible - to run main-less with only secondaries. However then you need to find out - which secondary took over the temporary role to be the main node. Look for - the `is_main_node` file in the fuzzer directories, eg. `sync-dir/hostname-*/is_main_node` - -It is *not* advisable to skip the synchronization script and run the fuzzers -directly on a network filesystem; unexpected latency and unkillable processes -in I/O wait state can mess things up. - -## 6) Remote monitoring and data collection - -You can use screen, nohup, tmux, or something equivalent to run remote -instances of afl-fuzz. If you redirect the program's output to a file, it will -automatically switch from a fancy UI to more limited status reports. There is -also basic machine-readable information which is always written to the -fuzzer_stats file in the output directory. Locally, that information can be -interpreted with afl-whatsup. - -In principle, you can use the status screen of the main (-M) instance to -monitor the overall fuzzing progress and decide when to stop. In this -mode, the most important signal is just that no new paths are being found -for a longer while. If you do not have a main instance, just pick any -single secondary instance to watch and go by that. - -You can also rely on that instance's output directory to collect the -synthesized corpus that covers all the noteworthy paths discovered anywhere -within the fleet. Secondary (-S) instances do not require any special -monitoring, other than just making sure that they are up. - -Keep in mind that crashing inputs are *not* automatically propagated to the -main instance, so you may still want to monitor for crashes fleet-wide -from within your synchronization or health checking scripts (see afl-whatsup). - -## 7) Asymmetric setups - -It is perhaps worth noting that all of the following is permitted: - - - Running afl-fuzz with conjunction with other guided tools that can extend - coverage (e.g., via concolic execution). Third-party tools simply need to - follow the protocol described above for pulling new test cases from - out_dir/<fuzzer_id>/queue/* and writing their own finds to sequentially - numbered id:nnnnnn files in out_dir/<ext_tool_id>/queue/*. - - - Running some of the synchronized fuzzers with different (but related) - target binaries. For example, simultaneously stress-testing several - different JPEG parsers (say, IJG jpeg and libjpeg-turbo) while sharing - the discovered test cases can have synergistic effects and improve the - overall coverage. - - (In this case, running one -M instance per target is necessary.) - - - Having some of the fuzzers invoke the binary in different ways. - For example, 'djpeg' supports several DCT modes, configurable with - a command-line flag, while 'dwebp' supports incremental and one-shot - decoding. In some scenarios, going after multiple distinct modes and then - pooling test cases will improve coverage. - - - Much less convincingly, running the synchronized fuzzers with different - starting test cases (e.g., progressive and standard JPEG) or dictionaries. - The synchronization mechanism ensures that the test sets will get fairly - homogeneous over time, but it introduces some initial variability. diff --git a/docs/perf_tips.md b/docs/perf_tips.md deleted file mode 100644 index 1e8fd4d0..00000000 --- a/docs/perf_tips.md +++ /dev/null @@ -1,209 +0,0 @@ -## Tips for performance optimization - - This file provides tips for troubleshooting slow or wasteful fuzzing jobs. - See README.md for the general instruction manual. - -## 1. Keep your test cases small - -This is probably the single most important step to take! Large test cases do -not merely take more time and memory to be parsed by the tested binary, but -also make the fuzzing process dramatically less efficient in several other -ways. - -To illustrate, let's say that you're randomly flipping bits in a file, one bit -at a time. Let's assume that if you flip bit #47, you will hit a security bug; -flipping any other bit just results in an invalid document. - -Now, if your starting test case is 100 bytes long, you will have a 71% chance of -triggering the bug within the first 1,000 execs - not bad! But if the test case -is 1 kB long, the probability that we will randomly hit the right pattern in -the same timeframe goes down to 11%. And if it has 10 kB of non-essential -cruft, the odds plunge to 1%. - -On top of that, with larger inputs, the binary may be now running 5-10x times -slower than before - so the overall drop in fuzzing efficiency may be easily -as high as 500x or so. - -In practice, this means that you shouldn't fuzz image parsers with your -vacation photos. Generate a tiny 16x16 picture instead, and run it through -`jpegtran` or `pngcrunch` for good measure. The same goes for most other types -of documents. - -There's plenty of small starting test cases in ../testcases/ - try them out -or submit new ones! - -If you want to start with a larger, third-party corpus, run `afl-cmin` with an -aggressive timeout on that data set first. - -## 2. Use a simpler target - -Consider using a simpler target binary in your fuzzing work. For example, for -image formats, bundled utilities such as `djpeg`, `readpng`, or `gifhisto` are -considerably (10-20x) faster than the convert tool from ImageMagick - all while exercising roughly the same library-level image parsing code. - -Even if you don't have a lightweight harness for a particular target, remember -that you can always use another, related library to generate a corpus that will -be then manually fed to a more resource-hungry program later on. - -Also note that reading the fuzzing input via stdin is faster than reading from -a file. - -## 3. Use LLVM persistent instrumentation - -The LLVM mode offers a "persistent", in-process fuzzing mode that can -work well for certain types of self-contained libraries, and for fast targets, -can offer performance gains up to 5-10x; and a "deferred fork server" mode -that can offer huge benefits for programs with high startup overhead. Both -modes require you to edit the source code of the fuzzed program, but the -changes often amount to just strategically placing a single line or two. - -If there are important data comparisons performed (e.g. `strcmp(ptr, MAGIC_HDR)`) -then using laf-intel (see instrumentation/README.laf-intel.md) will help `afl-fuzz` a lot -to get to the important parts in the code. - -If you are only interested in specific parts of the code being fuzzed, you can -instrument_files the files that are actually relevant. This improves the speed and -accuracy of afl. See instrumentation/README.instrument_list.md - -## 4. Profile and optimize the binary - -Check for any parameters or settings that obviously improve performance. For -example, the djpeg utility that comes with IJG jpeg and libjpeg-turbo can be -called with: - -```bash - -dct fast -nosmooth -onepass -dither none -scale 1/4 -``` - -...and that will speed things up. There is a corresponding drop in the quality -of decoded images, but it's probably not something you care about. - -In some programs, it is possible to disable output altogether, or at least use -an output format that is computationally inexpensive. For example, with image -transcoding tools, converting to a BMP file will be a lot faster than to PNG. - -With some laid-back parsers, enabling "strict" mode (i.e., bailing out after -first error) may result in smaller files and improved run time without -sacrificing coverage; for example, for sqlite, you may want to specify -bail. - -If the program is still too slow, you can use `strace -tt` or an equivalent -profiling tool to see if the targeted binary is doing anything silly. -Sometimes, you can speed things up simply by specifying `/dev/null` as the -config file, or disabling some compile-time features that aren't really needed -for the job (try `./configure --help`). One of the notoriously resource-consuming -things would be calling other utilities via `exec*()`, `popen()`, `system()`, or -equivalent calls; for example, tar can invoke external decompression tools -when it decides that the input file is a compressed archive. - -Some programs may also intentionally call `sleep()`, `usleep()`, or `nanosleep()`; -vim is a good example of that. Other programs may attempt `fsync()` and so on. -There are third-party libraries that make it easy to get rid of such code, -e.g.: - - https://launchpad.net/libeatmydata - -In programs that are slow due to unavoidable initialization overhead, you may -want to try the LLVM deferred forkserver mode (see README.llvm.md), -which can give you speed gains up to 10x, as mentioned above. - -Last but not least, if you are using ASAN and the performance is unacceptable, -consider turning it off for now, and manually examining the generated corpus -with an ASAN-enabled binary later on. - -## 5. Instrument just what you need - -Instrument just the libraries you actually want to stress-test right now, one -at a time. Let the program use system-wide, non-instrumented libraries for -any functionality you don't actually want to fuzz. For example, in most -cases, it doesn't make to instrument `libgmp` just because you're testing a -crypto app that relies on it for bignum math. - -Beware of programs that come with oddball third-party libraries bundled with -their source code (Spidermonkey is a good example of this). Check `./configure` -options to use non-instrumented system-wide copies instead. - -## 6. Parallelize your fuzzers - -The fuzzer is designed to need ~1 core per job. This means that on a, say, -4-core system, you can easily run four parallel fuzzing jobs with relatively -little performance hit. For tips on how to do that, see parallel_fuzzing.md. - -The `afl-gotcpu` utility can help you understand if you still have idle CPU -capacity on your system. (It won't tell you about memory bandwidth, cache -misses, or similar factors, but they are less likely to be a concern.) - -## 7. Keep memory use and timeouts in check - -Consider setting low values for `-m` and `-t`. - -For programs that are nominally very fast, but get sluggish for some inputs, -you can also try setting `-t` values that are more punishing than what `afl-fuzz` -dares to use on its own. On fast and idle machines, going down to `-t 5` may be -a viable plan. - -The `-m` parameter is worth looking at, too. Some programs can end up spending -a fair amount of time allocating and initializing megabytes of memory when -presented with pathological inputs. Low `-m` values can make them give up sooner -and not waste CPU time. - -## 8. Check OS configuration - -There are several OS-level factors that may affect fuzzing speed: - - - If you have no risk of power loss then run your fuzzing on a tmpfs - partition. This increases the performance noticably. - Alternatively you can use `AFL_TMPDIR` to point to a tmpfs location to - just write the input file to a tmpfs. - - High system load. Use idle machines where possible. Kill any non-essential - CPU hogs (idle browser windows, media players, complex screensavers, etc). - - Network filesystems, either used for fuzzer input / output, or accessed by - the fuzzed binary to read configuration files (pay special attention to the - home directory - many programs search it for dot-files). - - Disable all the spectre, meltdown etc. security countermeasures in the - kernel if your machine is properly separated: - -``` -ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off -no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable -nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off -spectre_v2=off stf_barrier=off -``` - In most Linux distributions you can put this into a `/etc/default/grub` - variable. - You can use `sudo afl-persistent-config` to set these options for you. - -The following list of changes are made when executing `afl-system-config`: - - - On-demand CPU scaling. The Linux `ondemand` governor performs its analysis - on a particular schedule and is known to underestimate the needs of - short-lived processes spawned by `afl-fuzz` (or any other fuzzer). On Linux, - this can be fixed with: - -``` bash - cd /sys/devices/system/cpu - echo performance | tee cpu*/cpufreq/scaling_governor -``` - - On other systems, the impact of CPU scaling will be different; when fuzzing, - use OS-specific tools to find out if all cores are running at full speed. - - Transparent huge pages. Some allocators, such as `jemalloc`, can incur a - heavy fuzzing penalty when transparent huge pages (THP) are enabled in the - kernel. You can disable this via: - -```bash - echo never > /sys/kernel/mm/transparent_hugepage/enabled -``` - - - Suboptimal scheduling strategies. The significance of this will vary from - one target to another, but on Linux, you may want to make sure that the - following options are set: - -```bash - echo 1 >/proc/sys/kernel/sched_child_runs_first - echo 1 >/proc/sys/kernel/sched_autogroup_enabled -``` - - Setting a different scheduling policy for the fuzzer process - say - `SCHED_RR` - can usually speed things up, too, but needs to be done with - care. - diff --git a/docs/sister_projects.md b/docs/sister_projects.md deleted file mode 100644 index 613bc778..00000000 --- a/docs/sister_projects.md +++ /dev/null @@ -1,319 +0,0 @@ -# Sister projects - -This doc lists some of the projects that are inspired by, derived from, -designed for, or meant to integrate with AFL. See README.md for the general -instruction manual. - -!!! -!!! This list is outdated and needs an update, missing: e.g. Angora, FairFuzz -!!! - -## Support for other languages / environments: - -### Python AFL (Jakub Wilk) - -Allows fuzz-testing of Python programs. Uses custom instrumentation and its -own forkserver. - -https://jwilk.net/software/python-afl - -### Go-fuzz (Dmitry Vyukov) - -AFL-inspired guided fuzzing approach for Go targets: - -https://github.com/dvyukov/go-fuzz - -### afl.rs (Keegan McAllister) - -Allows Rust features to be easily fuzzed with AFL (using the LLVM mode). - -https://github.com/kmcallister/afl.rs - -### OCaml support (KC Sivaramakrishnan) - -Adds AFL-compatible instrumentation to OCaml programs. - -https://github.com/ocamllabs/opam-repo-dev/pull/23 -https://canopy.mirage.io/Posts/Fuzzing - -### AFL for GCJ Java and other GCC frontends (-) - -GCC Java programs are actually supported out of the box - simply rename -afl-gcc to afl-gcj. Unfortunately, by default, unhandled exceptions in GCJ do -not result in abort() being called, so you will need to manually add a -top-level exception handler that exits with SIGABRT or something equivalent. - -Other GCC-supported languages should be fairly easy to get working, but may -face similar problems. See https://gcc.gnu.org/frontends.html for a list of -options. - -## AFL-style in-process fuzzer for LLVM (Kostya Serebryany) - -Provides an evolutionary instrumentation-guided fuzzing harness that allows -some programs to be fuzzed without the fork / execve overhead. (Similar -functionality is now available as the "persistent" feature described in -[the llvm_mode readme](../instrumentation/README.llvm.md)) - -https://llvm.org/docs/LibFuzzer.html - -## TriforceAFL (Tim Newsham and Jesse Hertz) - -Leverages QEMU full system emulation mode to allow AFL to target operating -systems and other alien worlds: - -https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2016/june/project-triforce-run-afl-on-everything/ - -## WinAFL (Ivan Fratric) - -As the name implies, allows you to fuzz Windows binaries (using DynamoRio). - -https://github.com/ivanfratric/winafl - -Another Windows alternative may be: - -https://github.com/carlosgprado/BrundleFuzz/ - -## Network fuzzing - -### Preeny (Yan Shoshitaishvili) - -Provides a fairly simple way to convince dynamically linked network-centric -programs to read from a file or not fork. Not AFL-specific, but described as -useful by many users. Some assembly required. - -https://github.com/zardus/preeny - -## Distributed fuzzing and related automation - -### roving (Richo Healey) - -A client-server architecture for effortlessly orchestrating AFL runs across -a fleet of machines. You don't want to use this on systems that face the -Internet or live in other untrusted environments. - -https://github.com/richo/roving - -### Distfuzz-AFL (Martijn Bogaard) - -Simplifies the management of afl-fuzz instances on remote machines. The -author notes that the current implementation isn't secure and should not -be exposed on the Internet. - -https://github.com/MartijnB/disfuzz-afl - -### AFLDFF (quantumvm) - -A nice GUI for managing AFL jobs. - -https://github.com/quantumvm/AFLDFF - -### afl-launch (Ben Nagy) - -Batch AFL launcher utility with a simple CLI. - -https://github.com/bnagy/afl-launch - -### AFL Utils (rc0r) - -Simplifies the triage of discovered crashes, start parallel instances, etc. - -https://github.com/rc0r/afl-utils - -### AFL crash analyzer (floyd) - -Another crash triage tool: - -https://github.com/floyd-fuh/afl-crash-analyzer - -### afl-extras (fekir) - -Collect data, parallel afl-tmin, startup scripts. - -https://github.com/fekir/afl-extras - -### afl-fuzzing-scripts (Tobias Ospelt) - -Simplifies starting up multiple parallel AFL jobs. - -https://github.com/floyd-fuh/afl-fuzzing-scripts/ - -### afl-sid (Jacek Wielemborek) - -Allows users to more conveniently build and deploy AFL via Docker. - -https://github.com/d33tah/afl-sid - -Another Docker-related project: - -https://github.com/ozzyjohnson/docker-afl - -### afl-monitor (Paul S. Ziegler) - -Provides more detailed and versatile statistics about your running AFL jobs. - -https://github.com/reflare/afl-monitor - -### FEXM (Security in Telecommunications) - -Fully automated fuzzing framework, based on AFL - -https://github.com/fgsect/fexm - -## Crash triage, coverage analysis, and other companion tools: - -### afl-crash-analyzer (Tobias Ospelt) - -Makes it easier to navigate and annotate crashing test cases. - -https://github.com/floyd-fuh/afl-crash-analyzer/ - -### Crashwalk (Ben Nagy) - -AFL-aware tool to annotate and sort through crashing test cases. - -https://github.com/bnagy/crashwalk - -### afl-cov (Michael Rash) - -Produces human-readable coverage data based on the output queue of afl-fuzz. - -https://github.com/mrash/afl-cov - -### afl-sancov (Bhargava Shastry) - -Similar to afl-cov, but uses clang sanitizer instrumentation. - -https://github.com/bshastry/afl-sancov - -### RecidiVM (Jakub Wilk) - -Makes it easy to estimate memory usage limits when fuzzing with ASAN or MSAN. - -https://jwilk.net/software/recidivm - -### aflize (Jacek Wielemborek) - -Automatically build AFL-enabled versions of Debian packages. - -https://github.com/d33tah/aflize - -### afl-ddmin-mod (Markus Teufelberger) - -A variant of afl-tmin that uses a more sophisticated (but slower) -minimization algorithm. - -https://github.com/MarkusTeufelberger/afl-ddmin-mod - -### afl-kit (Kuang-che Wu) - -Replacements for afl-cmin and afl-tmin with additional features, such -as the ability to filter crashes based on stderr patterns. - -https://github.com/kcwu/afl-kit - -## Narrow-purpose or experimental: - -### Cygwin support (Ali Rizvi-Santiago) - -Pretty self-explanatory. As per the author, this "mostly" ports AFL to -Windows. Field reports welcome! - -https://github.com/arizvisa/afl-cygwin - -### Pause and resume scripts (Ben Nagy) - -Simple automation to suspend and resume groups of fuzzing jobs. - -https://github.com/bnagy/afl-trivia - -### Static binary-only instrumentation (Aleksandar Nikolich) - -Allows black-box binaries to be instrumented statically (i.e., by modifying -the binary ahead of the time, rather than translating it on the run). Author -reports better performance compared to QEMU, but occasional translation -errors with stripped binaries. - -https://github.com/vanhauser-thc/afl-dyninst - -### AFL PIN (Parker Thompson) - -Early-stage Intel PIN instrumentation support (from before we settled on -faster-running QEMU). - -https://github.com/mothran/aflpin - -### AFL-style instrumentation in llvm (Kostya Serebryany) - -Allows AFL-equivalent instrumentation to be injected at compiler level. -This is currently not supported by AFL as-is, but may be useful in other -projects. - -https://code.google.com/p/address-sanitizer/wiki/AsanCoverage#Coverage_counters - -### AFL JS (Han Choongwoo) - -One-off optimizations to speed up the fuzzing of JavaScriptCore (now likely -superseded by LLVM deferred forkserver init - see README.llvm.md). - -https://github.com/tunz/afl-fuzz-js - -### AFL harness for fwknop (Michael Rash) - -An example of a fairly involved integration with AFL. - -https://github.com/mrash/fwknop/tree/master/test/afl - -### Building harnesses for DNS servers (Jonathan Foote, Ron Bowes) - -Two articles outlining the general principles and showing some example code. - -https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop -https://goo.gl/j9EgFf - -### Fuzzer shell for SQLite (Richard Hipp) - -A simple SQL shell designed specifically for fuzzing the underlying library. - -https://www.sqlite.org/src/artifact/9e7e273da2030371 - -### Support for Python mutation modules (Christian Holler) - -now integrated in AFL++, originally from here -https://github.com/choller/afl/blob/master/docs/mozilla/python_modules.txt - -### Support for selective instrumentation (Christian Holler) - -now integrated in AFL++, originally from here -https://github.com/choller/afl/blob/master/docs/mozilla/partial_instrumentation.txt - -### Syzkaller (Dmitry Vyukov) - -A similar guided approach as applied to fuzzing syscalls: - -https://github.com/google/syzkaller/wiki/Found-Bugs -https://github.com/dvyukov/linux/commit/33787098ffaaa83b8a7ccf519913ac5fd6125931 -https://events.linuxfoundation.org/sites/events/files/slides/AFL%20filesystem%20fuzzing%2C%20Vault%202016_0.pdf - - -### Kernel Snapshot Fuzzing using Unicornafl (Security in Telecommunications) - -https://github.com/fgsect/unicorefuzz - -### Android support (ele7enxxh) - -Based on a somewhat dated version of AFL: - -https://github.com/ele7enxxh/android-afl - -### CGI wrapper (floyd) - -Facilitates the testing of CGI scripts. - -https://github.com/floyd-fuh/afl-cgi-wrapper - -### Fuzzing difficulty estimation (Marcel Boehme) - -A fork of AFL that tries to quantify the likelihood of finding additional -paths or crashes at any point in a fuzzing job. - -https://github.com/mboehme/pythia diff --git a/docs/status_screen.md b/docs/status_screen.md deleted file mode 100644 index b1cb9696..00000000 --- a/docs/status_screen.md +++ /dev/null @@ -1,444 +0,0 @@ -# Understanding the status screen - -This document provides an overview of the status screen - plus tips for -troubleshooting any warnings and red text shown in the UI. See README.md for -the general instruction manual. - -## A note about colors - -The status screen and error messages use colors to keep things readable and -attract your attention to the most important details. For example, red almost -always means "consult this doc" :-) - -Unfortunately, the UI will render correctly only if your terminal is using -traditional un*x palette (white text on black background) or something close -to that. - -If you are using inverse video, you may want to change your settings, say: - -- For GNOME Terminal, go to `Edit > Profile` preferences, select the "colors" tab, and from the list of built-in schemes, choose "white on black". -- For the MacOS X Terminal app, open a new window using the "Pro" scheme via the `Shell > New Window` menu (or make "Pro" your default). - -Alternatively, if you really like your current colors, you can edit config.h -to comment out USE_COLORS, then do `make clean all`. - -I'm not aware of any other simple way to make this work without causing -other side effects - sorry about that. - -With that out of the way, let's talk about what's actually on the screen... - -### The status bar - -``` -american fuzzy lop ++3.01a (default) [fast] {0} -``` - -The top line shows you which mode afl-fuzz is running in -(normal: "american fuzy lop", crash exploration mode: "peruvian rabbit mode") -and the version of AFL++. -Next to the version is the banner, which, if not set with -T by hand, will -either show the binary name being fuzzed, or the -M/-S main/secondary name for -parallel fuzzing. -Second to last is the power schedule mode being run (default: fast). -Finally, the last item is the CPU id. - -### Process timing - -``` - +----------------------------------------------------+ - | run time : 0 days, 8 hrs, 32 min, 43 sec | - | last new path : 0 days, 0 hrs, 6 min, 40 sec | - | last uniq crash : none seen yet | - | last uniq hang : 0 days, 1 hrs, 24 min, 32 sec | - +----------------------------------------------------+ -``` - -This section is fairly self-explanatory: it tells you how long the fuzzer has -been running and how much time has elapsed since its most recent finds. This is -broken down into "paths" (a shorthand for test cases that trigger new execution -patterns), crashes, and hangs. - -When it comes to timing: there is no hard rule, but most fuzzing jobs should be -expected to run for days or weeks; in fact, for a moderately complex project, the -first pass will probably take a day or so. Every now and then, some jobs -will be allowed to run for months. - -There's one important thing to watch out for: if the tool is not finding new -paths within several minutes of starting, you're probably not invoking the -target binary correctly and it never gets to parse the input files we're -throwing at it; another possible explanations are that the default memory limit -(`-m`) is too restrictive, and the program exits after failing to allocate a -buffer very early on; or that the input files are patently invalid and always -fail a basic header check. - -If there are no new paths showing up for a while, you will eventually see a big -red warning in this section, too :-) - -### Overall results - -``` - +-----------------------+ - | cycles done : 0 | - | total paths : 2095 | - | uniq crashes : 0 | - | uniq hangs : 19 | - +-----------------------+ -``` - -The first field in this section gives you the count of queue passes done so far - that is, the number of times the fuzzer went over all the interesting test -cases discovered so far, fuzzed them, and looped back to the very beginning. -Every fuzzing session should be allowed to complete at least one cycle; and -ideally, should run much longer than that. - -As noted earlier, the first pass can take a day or longer, so sit back and -relax. - -To help make the call on when to hit `Ctrl-C`, the cycle counter is color-coded. -It is shown in magenta during the first pass, progresses to yellow if new finds -are still being made in subsequent rounds, then blue when that ends - and -finally, turns green after the fuzzer hasn't been seeing any action for a -longer while. - -The remaining fields in this part of the screen should be pretty obvious: -there's the number of test cases ("paths") discovered so far, and the number of -unique faults. The test cases, crashes, and hangs can be explored in real-time -by browsing the output directory, as discussed in README.md. - -### Cycle progress - -``` - +-------------------------------------+ - | now processing : 1296 (61.86%) | - | paths timed out : 0 (0.00%) | - +-------------------------------------+ -``` - -This box tells you how far along the fuzzer is with the current queue cycle: it -shows the ID of the test case it is currently working on, plus the number of -inputs it decided to ditch because they were persistently timing out. - -The "*" suffix sometimes shown in the first line means that the currently -processed path is not "favored" (a property discussed later on). - -### Map coverage - -``` - +--------------------------------------+ - | map density : 10.15% / 29.07% | - | count coverage : 4.03 bits/tuple | - +--------------------------------------+ -``` - -The section provides some trivia about the coverage observed by the -instrumentation embedded in the target binary. - -The first line in the box tells you how many branch tuples we have already -hit, in proportion to how much the bitmap can hold. The number on the left -describes the current input; the one on the right is the value for the entire -input corpus. - -Be wary of extremes: - - - Absolute numbers below 200 or so suggest one of three things: that the - program is extremely simple; that it is not instrumented properly (e.g., - due to being linked against a non-instrumented copy of the target - library); or that it is bailing out prematurely on your input test cases. - The fuzzer will try to mark this in pink, just to make you aware. - - Percentages over 70% may very rarely happen with very complex programs - that make heavy use of template-generated code. - Because high bitmap density makes it harder for the fuzzer to reliably - discern new program states, I recommend recompiling the binary with - `AFL_INST_RATIO=10` or so and trying again (see env_variables.md). - The fuzzer will flag high percentages in red. Chances are, you will never - see that unless you're fuzzing extremely hairy software (say, v8, perl, - ffmpeg). - -The other line deals with the variability in tuple hit counts seen in the -binary. In essence, if every taken branch is always taken a fixed number of -times for all the inputs we have tried, this will read `1.00`. As we manage -to trigger other hit counts for every branch, the needle will start to move -toward `8.00` (every bit in the 8-bit map hit), but will probably never -reach that extreme. - -Together, the values can be useful for comparing the coverage of several -different fuzzing jobs that rely on the same instrumented binary. - -### Stage progress - -``` - +-------------------------------------+ - | now trying : interest 32/8 | - | stage execs : 3996/34.4k (11.62%) | - | total execs : 27.4M | - | exec speed : 891.7/sec | - +-------------------------------------+ -``` - -This part gives you an in-depth peek at what the fuzzer is actually doing right -now. It tells you about the current stage, which can be any of: - - - calibration - a pre-fuzzing stage where the execution path is examined - to detect anomalies, establish baseline execution speed, and so on. Executed - very briefly whenever a new find is being made. - - trim L/S - another pre-fuzzing stage where the test case is trimmed to the - shortest form that still produces the same execution path. The length (L) - and stepover (S) are chosen in general relationship to file size. - - bitflip L/S - deterministic bit flips. There are L bits toggled at any given - time, walking the input file with S-bit increments. The current L/S variants - are: `1/1`, `2/1`, `4/1`, `8/8`, `16/8`, `32/8`. - - arith L/8 - deterministic arithmetics. The fuzzer tries to subtract or add - small integers to 8-, 16-, and 32-bit values. The stepover is always 8 bits. - - interest L/8 - deterministic value overwrite. The fuzzer has a list of known - "interesting" 8-, 16-, and 32-bit values to try. The stepover is 8 bits. - - extras - deterministic injection of dictionary terms. This can be shown as - "user" or "auto", depending on whether the fuzzer is using a user-supplied - dictionary (`-x`) or an auto-created one. You will also see "over" or "insert", - depending on whether the dictionary words overwrite existing data or are - inserted by offsetting the remaining data to accommodate their length. - - havoc - a sort-of-fixed-length cycle with stacked random tweaks. The - operations attempted during this stage include bit flips, overwrites with - random and "interesting" integers, block deletion, block duplication, plus - assorted dictionary-related operations (if a dictionary is supplied in the - first place). - - splice - a last-resort strategy that kicks in after the first full queue - cycle with no new paths. It is equivalent to 'havoc', except that it first - splices together two random inputs from the queue at some arbitrarily - selected midpoint. - - sync - a stage used only when `-M` or `-S` is set (see parallel_fuzzing.md). - No real fuzzing is involved, but the tool scans the output from other - fuzzers and imports test cases as necessary. The first time this is done, - it may take several minutes or so. - -The remaining fields should be fairly self-evident: there's the exec count -progress indicator for the current stage, a global exec counter, and a -benchmark for the current program execution speed. This may fluctuate from -one test case to another, but the benchmark should be ideally over 500 execs/sec -most of the time - and if it stays below 100, the job will probably take very -long. - -The fuzzer will explicitly warn you about slow targets, too. If this happens, -see the [perf_tips.md](perf_tips.md) file included with the fuzzer for ideas on how to speed -things up. - -### Findings in depth - -``` - +--------------------------------------+ - | favored paths : 879 (41.96%) | - | new edges on : 423 (20.19%) | - | total crashes : 0 (0 unique) | - | total tmouts : 24 (19 unique) | - +--------------------------------------+ -``` - -This gives you several metrics that are of interest mostly to complete nerds. -The section includes the number of paths that the fuzzer likes the most based -on a minimization algorithm baked into the code (these will get considerably -more air time), and the number of test cases that actually resulted in better -edge coverage (versus just pushing the branch hit counters up). There are also -additional, more detailed counters for crashes and timeouts. - -Note that the timeout counter is somewhat different from the hang counter; this -one includes all test cases that exceeded the timeout, even if they did not -exceed it by a margin sufficient to be classified as hangs. - -### Fuzzing strategy yields - -``` - +-----------------------------------------------------+ - | bit flips : 57/289k, 18/289k, 18/288k | - | byte flips : 0/36.2k, 4/35.7k, 7/34.6k | - | arithmetics : 53/2.54M, 0/537k, 0/55.2k | - | known ints : 8/322k, 12/1.32M, 10/1.70M | - | dictionary : 9/52k, 1/53k, 1/24k | - |havoc/splice : 1903/20.0M, 0/0 | - |py/custom/rq : unused, 53/2.54M, unused | - | trim/eff : 20.31%/9201, 17.05% | - +-----------------------------------------------------+ -``` - -This is just another nerd-targeted section keeping track of how many paths we -have netted, in proportion to the number of execs attempted, for each of the -fuzzing strategies discussed earlier on. This serves to convincingly validate -assumptions about the usefulness of the various approaches taken by afl-fuzz. - -The trim strategy stats in this section are a bit different than the rest. -The first number in this line shows the ratio of bytes removed from the input -files; the second one corresponds to the number of execs needed to achieve this -goal. Finally, the third number shows the proportion of bytes that, although -not possible to remove, were deemed to have no effect and were excluded from -some of the more expensive deterministic fuzzing steps. - -Note that when deterministic mutation mode is off (which is the default -because it is not very efficient) the first five lines display -"disabled (default, enable with -D)". - -Only what is activated will have counter shown. - -### Path geometry - -``` - +---------------------+ - | levels : 5 | - | pending : 1570 | - | pend fav : 583 | - | own finds : 0 | - | imported : 0 | - | stability : 100.00% | - +---------------------+ -``` - -The first field in this section tracks the path depth reached through the -guided fuzzing process. In essence: the initial test cases supplied by the -user are considered "level 1". The test cases that can be derived from that -through traditional fuzzing are considered "level 2"; the ones derived by -using these as inputs to subsequent fuzzing rounds are "level 3"; and so forth. -The maximum depth is therefore a rough proxy for how much value you're getting -out of the instrumentation-guided approach taken by afl-fuzz. - -The next field shows you the number of inputs that have not gone through any -fuzzing yet. The same stat is also given for "favored" entries that the fuzzer -really wants to get to in this queue cycle (the non-favored entries may have to -wait a couple of cycles to get their chance). - -Next, we have the number of new paths found during this fuzzing section and -imported from other fuzzer instances when doing parallelized fuzzing; and the -extent to which identical inputs appear to sometimes produce variable behavior -in the tested binary. - -That last bit is actually fairly interesting: it measures the consistency of -observed traces. If a program always behaves the same for the same input data, -it will earn a score of 100%. When the value is lower but still shown in purple, -the fuzzing process is unlikely to be negatively affected. If it goes into red, -you may be in trouble, since AFL will have difficulty discerning between -meaningful and "phantom" effects of tweaking the input file. - -Now, most targets will just get a 100% score, but when you see lower figures, -there are several things to look at: - - - The use of uninitialized memory in conjunction with some intrinsic sources - of entropy in the tested binary. Harmless to AFL, but could be indicative - of a security bug. - - Attempts to manipulate persistent resources, such as left over temporary - files or shared memory objects. This is usually harmless, but you may want - to double-check to make sure the program isn't bailing out prematurely. - Running out of disk space, SHM handles, or other global resources can - trigger this, too. - - Hitting some functionality that is actually designed to behave randomly. - Generally harmless. For example, when fuzzing sqlite, an input like - `select random();` will trigger a variable execution path. - - Multiple threads executing at once in semi-random order. This is harmless - when the 'stability' metric stays over 90% or so, but can become an issue - if not. Here's what to try: - * Use afl-clang-fast from [instrumentation](../instrumentation/) - it uses a thread-local tracking - model that is less prone to concurrency issues, - * See if the target can be compiled or run without threads. Common - `./configure` options include `--without-threads`, `--disable-pthreads`, or - `--disable-openmp`. - * Replace pthreads with GNU Pth (https://www.gnu.org/software/pth/), which - allows you to use a deterministic scheduler. - - In persistent mode, minor drops in the "stability" metric can be normal, - because not all the code behaves identically when re-entered; but major - dips may signify that the code within `__AFL_LOOP()` is not behaving - correctly on subsequent iterations (e.g., due to incomplete clean-up or - reinitialization of the state) and that most of the fuzzing effort goes - to waste. - -The paths where variable behavior is detected are marked with a matching entry -in the `<out_dir>/queue/.state/variable_behavior/` directory, so you can look -them up easily. - -### CPU load - -``` - [cpu: 25%] -``` - -This tiny widget shows the apparent CPU utilization on the local system. It is -calculated by taking the number of processes in the "runnable" state, and then -comparing it to the number of logical cores on the system. - -If the value is shown in green, you are using fewer CPU cores than available on -your system and can probably parallelize to improve performance; for tips on -how to do that, see parallel_fuzzing.md. - -If the value is shown in red, your CPU is *possibly* oversubscribed, and -running additional fuzzers may not give you any benefits. - -Of course, this benchmark is very simplistic; it tells you how many processes -are ready to run, but not how resource-hungry they may be. It also doesn't -distinguish between physical cores, logical cores, and virtualized CPUs; the -performance characteristics of each of these will differ quite a bit. - -If you want a more accurate measurement, you can run the `afl-gotcpu` utility from the command line. - -### Addendum: status and plot files - -For unattended operation, some of the key status screen information can be also -found in a machine-readable format in the fuzzer_stats file in the output -directory. This includes: - - - `start_time` - unix time indicating the start time of afl-fuzz - - `last_update` - unix time corresponding to the last update of this file - - `run_time` - run time in seconds to the last update of this file - - `fuzzer_pid` - PID of the fuzzer process - - `cycles_done` - queue cycles completed so far - - `cycles_wo_finds` - number of cycles without any new paths found - - `execs_done` - number of execve() calls attempted - - `execs_per_sec` - overall number of execs per second - - `paths_total` - total number of entries in the queue - - `paths_favored` - number of queue entries that are favored - - `paths_found` - number of entries discovered through local fuzzing - - `paths_imported` - number of entries imported from other instances - - `max_depth` - number of levels in the generated data set - - `cur_path` - currently processed entry number - - `pending_favs` - number of favored entries still waiting to be fuzzed - - `pending_total` - number of all entries waiting to be fuzzed - - `variable_paths` - number of test cases showing variable behavior - - `stability` - percentage of bitmap bytes that behave consistently - - `bitmap_cvg` - percentage of edge coverage found in the map so far - - `unique_crashes` - number of unique crashes recorded - - `unique_hangs` - number of unique hangs encountered - - `last_path` - seconds since the last path was found - - `last_crash` - seconds since the last crash was found - - `last_hang` - seconds since the last hang was found - - `execs_since_crash` - execs since the last crash was found - - `exec_timeout` - the -t command line value - - `slowest_exec_ms` - real time of the slowest execution in ms - - `peak_rss_mb` - max rss usage reached during fuzzing in MB - - `edges_found` - how many edges have been found - - `var_byte_count` - how many edges are non-deterministic - - `afl_banner` - banner text (e.g. the target name) - - `afl_version` - the version of AFL used - - `target_mode` - default, persistent, qemu, unicorn, non-instrumented - - `command_line` - full command line used for the fuzzing session - -Most of these map directly to the UI elements discussed earlier on. - -On top of that, you can also find an entry called `plot_data`, containing a -plottable history for most of these fields. If you have gnuplot installed, you -can turn this into a nice progress report with the included `afl-plot` tool. - - -### Addendum: Automatically send metrics with StatsD - -In a CI environment or when running multiple fuzzers, it can be tedious to -log into each of them or deploy scripts to read the fuzzer statistics. -Using `AFL_STATSD` (and the other related environment variables `AFL_STATSD_HOST`, -`AFL_STATSD_PORT`, `AFL_STATSD_TAGS_FLAVOR`) you can automatically send metrics -to your favorite StatsD server. Depending on your StatsD server you will be able -to monitor, trigger alerts or perform actions based on these metrics (e.g: alert on -slow exec/s for a new build, threshold of crashes, time since last crash > X, etc). - -The selected metrics are a subset of all the metrics found in the status and in -the plot file. The list is the following: `cycle_done`, `cycles_wo_finds`, -`execs_done`,`execs_per_sec`, `paths_total`, `paths_favored`, `paths_found`, -`paths_imported`, `max_depth`, `cur_path`, `pending_favs`, `pending_total`, -`variable_paths`, `unique_crashes`, `unique_hangs`, `total_crashes`, -`slowest_exec_ms`, `edges_found`, `var_byte_count`, `havoc_expansion`. -Their definitions can be found in the addendum above. - -When using multiple fuzzer instances with StatsD it is *strongly* recommended to setup -the flavor (AFL_STATSD_TAGS_FLAVOR) to match your StatsD server. This will allow you -to see individual fuzzer performance, detect bad ones, see the progress of each -strategy... diff --git a/docs/technical_details.md b/docs/technical_details.md deleted file mode 100644 index 994ffe9f..00000000 --- a/docs/technical_details.md +++ /dev/null @@ -1,550 +0,0 @@ -# Technical "whitepaper" for afl-fuzz - - -NOTE: this document is mostly outdated! - - -This document provides a quick overview of the guts of American Fuzzy Lop. -See README.md for the general instruction manual; and for a discussion of -motivations and design goals behind AFL, see historical_notes.md. - -## 0. Design statement - -American Fuzzy Lop does its best not to focus on any singular principle of -operation and not be a proof-of-concept for any specific theory. The tool can -be thought of as a collection of hacks that have been tested in practice, -found to be surprisingly effective, and have been implemented in the simplest, -most robust way I could think of at the time. - -Many of the resulting features are made possible thanks to the availability of -lightweight instrumentation that served as a foundation for the tool, but this -mechanism should be thought of merely as a means to an end. The only true -governing principles are speed, reliability, and ease of use. - -## 1. Coverage measurements - -The instrumentation injected into compiled programs captures branch (edge) -coverage, along with coarse branch-taken hit counts. The code injected at -branch points is essentially equivalent to: - -```c - cur_location = <COMPILE_TIME_RANDOM>; - shared_mem[cur_location ^ prev_location]++; - prev_location = cur_location >> 1; -``` - -The `cur_location` value is generated randomly to simplify the process of -linking complex projects and keep the XOR output distributed uniformly. - -The `shared_mem[]` array is a 64 kB SHM region passed to the instrumented binary -by the caller. Every byte set in the output map can be thought of as a hit for -a particular (`branch_src`, `branch_dst`) tuple in the instrumented code. - -The size of the map is chosen so that collisions are sporadic with almost all -of the intended targets, which usually sport between 2k and 10k discoverable -branch points: - -``` - Branch cnt | Colliding tuples | Example targets - ------------+------------------+----------------- - 1,000 | 0.75% | giflib, lzo - 2,000 | 1.5% | zlib, tar, xz - 5,000 | 3.5% | libpng, libwebp - 10,000 | 7% | libxml - 20,000 | 14% | sqlite - 50,000 | 30% | - -``` - -At the same time, its size is small enough to allow the map to be analyzed -in a matter of microseconds on the receiving end, and to effortlessly fit -within L2 cache. - -This form of coverage provides considerably more insight into the execution -path of the program than simple block coverage. In particular, it trivially -distinguishes between the following execution traces: - -``` - A -> B -> C -> D -> E (tuples: AB, BC, CD, DE) - A -> B -> D -> C -> E (tuples: AB, BD, DC, CE) -``` - -This aids the discovery of subtle fault conditions in the underlying code, -because security vulnerabilities are more often associated with unexpected -or incorrect state transitions than with merely reaching a new basic block. - -The reason for the shift operation in the last line of the pseudocode shown -earlier in this section is to preserve the directionality of tuples (without -this, A ^ B would be indistinguishable from B ^ A) and to retain the identity -of tight loops (otherwise, A ^ A would be obviously equal to B ^ B). - -The absence of simple saturating arithmetic opcodes on Intel CPUs means that -the hit counters can sometimes wrap around to zero. Since this is a fairly -unlikely and localized event, it's seen as an acceptable performance trade-off. - -### 2. Detecting new behaviors - -The fuzzer maintains a global map of tuples seen in previous executions; this -data can be rapidly compared with individual traces and updated in just a couple -of dword- or qword-wide instructions and a simple loop. - -When a mutated input produces an execution trace containing new tuples, the -corresponding input file is preserved and routed for additional processing -later on (see section #3). Inputs that do not trigger new local-scale state -transitions in the execution trace (i.e., produce no new tuples) are discarded, -even if their overall control flow sequence is unique. - -This approach allows for a very fine-grained and long-term exploration of -program state while not having to perform any computationally intensive and -fragile global comparisons of complex execution traces, and while avoiding the -scourge of path explosion. - -To illustrate the properties of the algorithm, consider that the second trace -shown below would be considered substantially new because of the presence of -new tuples (CA, AE): - -``` - #1: A -> B -> C -> D -> E - #2: A -> B -> C -> A -> E -``` - -At the same time, with #2 processed, the following pattern will not be seen -as unique, despite having a markedly different overall execution path: - -``` - #3: A -> B -> C -> A -> B -> C -> A -> B -> C -> D -> E -``` - -In addition to detecting new tuples, the fuzzer also considers coarse tuple -hit counts. These are divided into several buckets: - -``` - 1, 2, 3, 4-7, 8-15, 16-31, 32-127, 128+ -``` - -To some extent, the number of buckets is an implementation artifact: it allows -an in-place mapping of an 8-bit counter generated by the instrumentation to -an 8-position bitmap relied on by the fuzzer executable to keep track of the -already-seen execution counts for each tuple. - -Changes within the range of a single bucket are ignored; transition from one -bucket to another is flagged as an interesting change in program control flow, -and is routed to the evolutionary process outlined in the section below. - -The hit count behavior provides a way to distinguish between potentially -interesting control flow changes, such as a block of code being executed -twice when it was normally hit only once. At the same time, it is fairly -insensitive to empirically less notable changes, such as a loop going from -47 cycles to 48. The counters also provide some degree of "accidental" -immunity against tuple collisions in dense trace maps. - -The execution is policed fairly heavily through memory and execution time -limits; by default, the timeout is set at 5x the initially-calibrated -execution speed, rounded up to 20 ms. The aggressive timeouts are meant to -prevent dramatic fuzzer performance degradation by descending into tarpits -that, say, improve coverage by 1% while being 100x slower; we pragmatically -reject them and hope that the fuzzer will find a less expensive way to reach -the same code. Empirical testing strongly suggests that more generous time -limits are not worth the cost. - -## 3. Evolving the input queue - -Mutated test cases that produced new state transitions within the program are -added to the input queue and used as a starting point for future rounds of -fuzzing. They supplement, but do not automatically replace, existing finds. - -In contrast to more greedy genetic algorithms, this approach allows the tool -to progressively explore various disjoint and possibly mutually incompatible -features of the underlying data format, as shown in this image: - -  - -Several practical examples of the results of this algorithm are discussed -here: - - https://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thin-air.html - https://lcamtuf.blogspot.com/2014/11/afl-fuzz-nobody-expects-cdata-sections.html - -The synthetic corpus produced by this process is essentially a compact -collection of "hmm, this does something new!" input files, and can be used to -seed any other testing processes down the line (for example, to manually -stress-test resource-intensive desktop apps). - -With this approach, the queue for most targets grows to somewhere between 1k -and 10k entries; approximately 10-30% of this is attributable to the discovery -of new tuples, and the remainder is associated with changes in hit counts. - -The following table compares the relative ability to discover file syntax and -explore program states when using several different approaches to guided -fuzzing. The instrumented target was GNU patch 2.7k.3 compiled with `-O3` and -seeded with a dummy text file; the session consisted of a single pass over the -input queue with afl-fuzz: - -``` - Fuzzer guidance | Blocks | Edges | Edge hit | Highest-coverage - strategy used | reached | reached | cnt var | test case generated - ------------------+---------+---------+----------+--------------------------- - (Initial file) | 156 | 163 | 1.00 | (none) - | | | | - Blind fuzzing S | 182 | 205 | 2.23 | First 2 B of RCS diff - Blind fuzzing L | 228 | 265 | 2.23 | First 4 B of -c mode diff - Block coverage | 855 | 1,130 | 1.57 | Almost-valid RCS diff - Edge coverage | 1,452 | 2,070 | 2.18 | One-chunk -c mode diff - AFL model | 1,765 | 2,597 | 4.99 | Four-chunk -c mode diff -``` - -The first entry for blind fuzzing ("S") corresponds to executing just a single -round of testing; the second set of figures ("L") shows the fuzzer running in a -loop for a number of execution cycles comparable with that of the instrumented -runs, which required more time to fully process the growing queue. - -Roughly similar results have been obtained in a separate experiment where the -fuzzer was modified to compile out all the random fuzzing stages and leave just -a series of rudimentary, sequential operations such as walking bit flips. -Because this mode would be incapable of altering the size of the input file, -the sessions were seeded with a valid unified diff: - -``` - Queue extension | Blocks | Edges | Edge hit | Number of unique - strategy used | reached | reached | cnt var | crashes found - ------------------+---------+---------+----------+------------------ - (Initial file) | 624 | 717 | 1.00 | - - | | | | - Blind fuzzing | 1,101 | 1,409 | 1.60 | 0 - Block coverage | 1,255 | 1,649 | 1.48 | 0 - Edge coverage | 1,259 | 1,734 | 1.72 | 0 - AFL model | 1,452 | 2,040 | 3.16 | 1 -``` - -At noted earlier on, some of the prior work on genetic fuzzing relied on -maintaining a single test case and evolving it to maximize coverage. At least -in the tests described above, this "greedy" approach appears to confer no -substantial benefits over blind fuzzing strategies. - -### 4. Culling the corpus - -The progressive state exploration approach outlined above means that some of -the test cases synthesized later on in the game may have edge coverage that -is a strict superset of the coverage provided by their ancestors. - -To optimize the fuzzing effort, AFL periodically re-evaluates the queue using a -fast algorithm that selects a smaller subset of test cases that still cover -every tuple seen so far, and whose characteristics make them particularly -favorable to the tool. - -The algorithm works by assigning every queue entry a score proportional to its -execution latency and file size; and then selecting lowest-scoring candidates -for each tuple. - -The tuples are then processed sequentially using a simple workflow: - - 1) Find next tuple not yet in the temporary working set, - 2) Locate the winning queue entry for this tuple, - 3) Register *all* tuples present in that entry's trace in the working set, - 4) Go to #1 if there are any missing tuples in the set. - -The generated corpus of "favored" entries is usually 5-10x smaller than the -starting data set. Non-favored entries are not discarded, but they are skipped -with varying probabilities when encountered in the queue: - - - If there are new, yet-to-be-fuzzed favorites present in the queue, 99% - of non-favored entries will be skipped to get to the favored ones. - - If there are no new favorites: - * If the current non-favored entry was fuzzed before, it will be skipped - 95% of the time. - * If it hasn't gone through any fuzzing rounds yet, the odds of skipping - drop down to 75%. - -Based on empirical testing, this provides a reasonable balance between queue -cycling speed and test case diversity. - -Slightly more sophisticated but much slower culling can be performed on input -or output corpora with `afl-cmin`. This tool permanently discards the redundant -entries and produces a smaller corpus suitable for use with `afl-fuzz` or -external tools. - -## 5. Trimming input files - -File size has a dramatic impact on fuzzing performance, both because large -files make the target binary slower, and because they reduce the likelihood -that a mutation would touch important format control structures, rather than -redundant data blocks. This is discussed in more detail in perf_tips.md. - -The possibility that the user will provide a low-quality starting corpus aside, -some types of mutations can have the effect of iteratively increasing the size -of the generated files, so it is important to counter this trend. - -Luckily, the instrumentation feedback provides a simple way to automatically -trim down input files while ensuring that the changes made to the files have no -impact on the execution path. - -The built-in trimmer in afl-fuzz attempts to sequentially remove blocks of data -with variable length and stepover; any deletion that doesn't affect the checksum -of the trace map is committed to disk. The trimmer is not designed to be -particularly thorough; instead, it tries to strike a balance between precision -and the number of `execve()` calls spent on the process, selecting the block size -and stepover to match. The average per-file gains are around 5-20%. - -The standalone `afl-tmin` tool uses a more exhaustive, iterative algorithm, and -also attempts to perform alphabet normalization on the trimmed files. The -operation of `afl-tmin` is as follows. - -First, the tool automatically selects the operating mode. If the initial input -crashes the target binary, afl-tmin will run in non-instrumented mode, simply -keeping any tweaks that produce a simpler file but still crash the target. -The same mode is used for hangs, if `-H` (hang mode) is specified. -If the target is non-crashing, the tool uses an instrumented mode and keeps only -the tweaks that produce exactly the same execution path. - -The actual minimization algorithm is: - - 1) Attempt to zero large blocks of data with large stepovers. Empirically, - this is shown to reduce the number of execs by preempting finer-grained - efforts later on. - 2) Perform a block deletion pass with decreasing block sizes and stepovers, - binary-search-style. - 3) Perform alphabet normalization by counting unique characters and trying - to bulk-replace each with a zero value. - 4) As a last result, perform byte-by-byte normalization on non-zero bytes. - -Instead of zeroing with a 0x00 byte, `afl-tmin` uses the ASCII digit '0'. This -is done because such a modification is much less likely to interfere with -text parsing, so it is more likely to result in successful minimization of -text files. - -The algorithm used here is less involved than some other test case -minimization approaches proposed in academic work, but requires far fewer -executions and tends to produce comparable results in most real-world -applications. - -## 6. Fuzzing strategies - -The feedback provided by the instrumentation makes it easy to understand the -value of various fuzzing strategies and optimize their parameters so that they -work equally well across a wide range of file types. The strategies used by -afl-fuzz are generally format-agnostic and are discussed in more detail here: - - https://lcamtuf.blogspot.com/2014/08/binary-fuzzing-strategies-what-works.html - -It is somewhat notable that especially early on, most of the work done by -`afl-fuzz` is actually highly deterministic, and progresses to random stacked -modifications and test case splicing only at a later stage. The deterministic -strategies include: - - - Sequential bit flips with varying lengths and stepovers, - - Sequential addition and subtraction of small integers, - - Sequential insertion of known interesting integers (`0`, `1`, `INT_MAX`, etc), - -The purpose of opening with deterministic steps is related to their tendency to -produce compact test cases and small diffs between the non-crashing and crashing -inputs. - -With deterministic fuzzing out of the way, the non-deterministic steps include -stacked bit flips, insertions, deletions, arithmetics, and splicing of different -test cases. - -The relative yields and `execve()` costs of all these strategies have been -investigated and are discussed in the aforementioned blog post. - -For the reasons discussed in historical_notes.md (chiefly, performance, -simplicity, and reliability), AFL generally does not try to reason about the -relationship between specific mutations and program states; the fuzzing steps -are nominally blind, and are guided only by the evolutionary design of the -input queue. - -That said, there is one (trivial) exception to this rule: when a new queue -entry goes through the initial set of deterministic fuzzing steps, and tweaks to -some regions in the file are observed to have no effect on the checksum of the -execution path, they may be excluded from the remaining phases of -deterministic fuzzing - and the fuzzer may proceed straight to random tweaks. -Especially for verbose, human-readable data formats, this can reduce the number -of execs by 10-40% or so without an appreciable drop in coverage. In extreme -cases, such as normally block-aligned tar archives, the gains can be as high as -90%. - -Because the underlying "effector maps" are local every queue entry and remain -in force only during deterministic stages that do not alter the size or the -general layout of the underlying file, this mechanism appears to work very -reliably and proved to be simple to implement. - -## 7. Dictionaries - -The feedback provided by the instrumentation makes it easy to automatically -identify syntax tokens in some types of input files, and to detect that certain -combinations of predefined or auto-detected dictionary terms constitute a -valid grammar for the tested parser. - -A discussion of how these features are implemented within afl-fuzz can be found -here: - - https://lcamtuf.blogspot.com/2015/01/afl-fuzz-making-up-grammar-with.html - -In essence, when basic, typically easily-obtained syntax tokens are combined -together in a purely random manner, the instrumentation and the evolutionary -design of the queue together provide a feedback mechanism to differentiate -between meaningless mutations and ones that trigger new behaviors in the -instrumented code - and to incrementally build more complex syntax on top of -this discovery. - -The dictionaries have been shown to enable the fuzzer to rapidly reconstruct -the grammar of highly verbose and complex languages such as JavaScript, SQL, -or XML; several examples of generated SQL statements are given in the blog -post mentioned above. - -Interestingly, the AFL instrumentation also allows the fuzzer to automatically -isolate syntax tokens already present in an input file. It can do so by looking -for run of bytes that, when flipped, produce a consistent change to the -program's execution path; this is suggestive of an underlying atomic comparison -to a predefined value baked into the code. The fuzzer relies on this signal -to build compact "auto dictionaries" that are then used in conjunction with -other fuzzing strategies. - -## 8. De-duping crashes - -De-duplication of crashes is one of the more important problems for any -competent fuzzing tool. Many of the naive approaches run into problems; in -particular, looking just at the faulting address may lead to completely -unrelated issues being clustered together if the fault happens in a common -library function (say, `strcmp`, `strcpy`); while checksumming call stack -backtraces can lead to extreme crash count inflation if the fault can be -reached through a number of different, possibly recursive code paths. - -The solution implemented in `afl-fuzz` considers a crash unique if any of two -conditions are met: - - - The crash trace includes a tuple not seen in any of the previous crashes, - - The crash trace is missing a tuple that was always present in earlier - faults. - -The approach is vulnerable to some path count inflation early on, but exhibits -a very strong self-limiting effect, similar to the execution path analysis -logic that is the cornerstone of `afl-fuzz`. - -## 9. Investigating crashes - -The exploitability of many types of crashes can be ambiguous; afl-fuzz tries -to address this by providing a crash exploration mode where a known-faulting -test case is fuzzed in a manner very similar to the normal operation of the -fuzzer, but with a constraint that causes any non-crashing mutations to be -thrown away. - -A detailed discussion of the value of this approach can be found here: - - https://lcamtuf.blogspot.com/2014/11/afl-fuzz-crash-exploration-mode.html - -The method uses instrumentation feedback to explore the state of the crashing -program to get past the ambiguous faulting condition and then isolate the -newly-found inputs for human review. - -On the subject of crashes, it is worth noting that in contrast to normal -queue entries, crashing inputs are *not* trimmed; they are kept exactly as -discovered to make it easier to compare them to the parent, non-crashing entry -in the queue. That said, `afl-tmin` can be used to shrink them at will. - -## 10 The fork server - -To improve performance, `afl-fuzz` uses a "fork server", where the fuzzed process -goes through `execve()`, linking, and libc initialization only once, and is then -cloned from a stopped process image by leveraging copy-on-write. The -implementation is described in more detail here: - - https://lcamtuf.blogspot.com/2014/10/fuzzing-binaries-without-execve.html - -The fork server is an integral aspect of the injected instrumentation and -simply stops at the first instrumented function to await commands from -`afl-fuzz`. - -With fast targets, the fork server can offer considerable performance gains, -usually between 1.5x and 2x. It is also possible to: - - - Use the fork server in manual ("deferred") mode, skipping over larger, - user-selected chunks of initialization code. It requires very modest - code changes to the targeted program, and With some targets, can - produce 10x+ performance gains. - - Enable "persistent" mode, where a single process is used to try out - multiple inputs, greatly limiting the overhead of repetitive `fork()` - calls. This generally requires some code changes to the targeted program, - but can improve the performance of fast targets by a factor of 5 or more - approximating the benefits of in-process fuzzing jobs while still - maintaining very robust isolation between the fuzzer process and the - targeted binary. - -## 11. Parallelization - -The parallelization mechanism relies on periodically examining the queues -produced by independently-running instances on other CPU cores or on remote -machines, and then selectively pulling in the test cases that, when tried -out locally, produce behaviors not yet seen by the fuzzer at hand. - -This allows for extreme flexibility in fuzzer setup, including running synced -instances against different parsers of a common data format, often with -synergistic effects. - -For more information about this design, see parallel_fuzzing.md. - -## 12. Binary-only instrumentation - -Instrumentation of black-box, binary-only targets is accomplished with the -help of a separately-built version of QEMU in "user emulation" mode. This also -allows the execution of cross-architecture code - say, ARM binaries on x86. - -QEMU uses basic blocks as translation units; the instrumentation is implemented -on top of this and uses a model roughly analogous to the compile-time hooks: - -```c - if (block_address > elf_text_start && block_address < elf_text_end) { - - cur_location = (block_address >> 4) ^ (block_address << 8); - shared_mem[cur_location ^ prev_location]++; - prev_location = cur_location >> 1; - - } -``` - -The shift-and-XOR-based scrambling in the second line is used to mask the -effects of instruction alignment. - -The start-up of binary translators such as QEMU, DynamoRIO, and PIN is fairly -slow; to counter this, the QEMU mode leverages a fork server similar to that -used for compiler-instrumented code, effectively spawning copies of an -already-initialized process paused at `_start`. - -First-time translation of a new basic block also incurs substantial latency. To -eliminate this problem, the AFL fork server is extended by providing a channel -between the running emulator and the parent process. The channel is used -to notify the parent about the addresses of any newly-encountered blocks and to -add them to the translation cache that will be replicated for future child -processes. - -As a result of these two optimizations, the overhead of the QEMU mode is -roughly 2-5x, compared to 100x+ for PIN. - -## 13. The `afl-analyze` tool - -The file format analyzer is a simple extension of the minimization algorithm -discussed earlier on; instead of attempting to remove no-op blocks, the tool -performs a series of walking byte flips and then annotates runs of bytes -in the input file. - -It uses the following classification scheme: - - - "No-op blocks" - segments where bit flips cause no apparent changes to - control flow. Common examples may be comment sections, pixel data within - a bitmap file, etc. - - "Superficial content" - segments where some, but not all, bitflips - produce some control flow changes. Examples may include strings in rich - documents (e.g., XML, RTF). - - "Critical stream" - a sequence of bytes where all bit flips alter control - flow in different but correlated ways. This may be compressed data, - non-atomically compared keywords or magic values, etc. - - "Suspected length field" - small, atomic integer that, when touched in - any way, causes a consistent change to program control flow, suggestive - of a failed length check. - - "Suspected cksum or magic int" - an integer that behaves similarly to a - length field, but has a numerical value that makes the length explanation - unlikely. This is suggestive of a checksum or other "magic" integer. - - "Suspected checksummed block" - a long block of data where any change - always triggers the same new execution path. Likely caused by failing - a checksum or a similar integrity check before any subsequent parsing - takes place. - - "Magic value section" - a generic token where changes cause the type - of binary behavior outlined earlier, but that doesn't meet any of the - other criteria. May be an atomically compared keyword or so. diff --git a/docs/third_party_tools.md b/docs/third_party_tools.md new file mode 100644 index 00000000..92229e84 --- /dev/null +++ b/docs/third_party_tools.md @@ -0,0 +1,57 @@ +# Tools that help fuzzing with AFL++ + +Speeding up fuzzing: +* [libfiowrapper](https://github.com/marekzmyslowski/libfiowrapper) - if the + function you want to fuzz requires loading a file, this allows using the + shared memory test case feature :-) - recommended. + +Minimization of test cases: +* [afl-pytmin](https://github.com/ilsani/afl-pytmin) - a wrapper for afl-tmin + that tries to speed up the process of minimization of a single test case by + using many CPU cores. +* [afl-ddmin-mod](https://github.com/MarkusTeufelberger/afl-ddmin-mod) - a + variation of afl-tmin based on the ddmin algorithm. +* [halfempty](https://github.com/googleprojectzero/halfempty) - is a fast + utility for minimizing test cases by Tavis Ormandy based on parallelization. + +Distributed execution: +* [disfuzz-afl](https://github.com/MartijnB/disfuzz-afl) - distributed fuzzing + for AFL. +* [AFLDFF](https://github.com/quantumvm/AFLDFF) - AFL distributed fuzzing + framework. +* [afl-launch](https://github.com/bnagy/afl-launch) - a tool for the execution + of many AFL instances. +* [afl-mothership](https://github.com/afl-mothership/afl-mothership) - + management and execution of many synchronized AFL fuzzers on AWS cloud. +* [afl-in-the-cloud](https://github.com/abhisek/afl-in-the-cloud) - another + script for running AFL in AWS. + +Deployment, management, monitoring, reporting +* [afl-utils](https://gitlab.com/rc0r/afl-utils) - a set of utilities for + automatic processing/analysis of crashes and reducing the number of test + cases. +* [afl-other-arch](https://github.com/shellphish/afl-other-arch) - is a set of + patches and scripts for easily adding support for various non-x86 + architectures for AFL. +* [afl-trivia](https://github.com/bnagy/afl-trivia) - a few small scripts to + simplify the management of AFL. +* [afl-monitor](https://github.com/reflare/afl-monitor) - a script for + monitoring AFL. +* [afl-manager](https://github.com/zx1340/afl-manager) - a web server on Python + for managing multi-afl. +* [afl-remote](https://github.com/block8437/afl-remote) - a web server for the + remote management of AFL instances. +* [afl-extras](https://github.com/fekir/afl-extras) - shell scripts to + parallelize afl-tmin, startup, and data collection. + +Crash processing +* [afl-crash-analyzer](https://github.com/floyd-fuh/afl-crash-analyzer) - + another crash analyzer for AFL. +* [fuzzer-utils](https://github.com/ThePatrickStar/fuzzer-utils) - a set of + scripts for the analysis of results. +* [atriage](https://github.com/Ayrx/atriage) - a simple triage tool. +* [afl-kit](https://github.com/kcwu/afl-kit) - afl-cmin on Python. +* [AFLize](https://github.com/d33tah/aflize) - a tool that automatically + generates builds of debian packages suitable for AFL. +* [afl-fid](https://github.com/FoRTE-Research/afl-fid) - a set of tools for + working with input data. \ No newline at end of file diff --git a/docs/tools.md b/docs/tools.md deleted file mode 100644 index ba96d0ce..00000000 --- a/docs/tools.md +++ /dev/null @@ -1,33 +0,0 @@ -# Tools that help fuzzing with AFL++ - -Speeding up fuzzing: - * [libfiowrapper](https://github.com/marekzmyslowski/libfiowrapper) - if the function you want to fuzz requires loading a file, this allows using the shared memory testcase feature :-) - recommended. - -Minimization of test cases: - * [afl-pytmin](https://github.com/ilsani/afl-pytmin) - a wrapper for afl-tmin that tries to speed up the process of minimization of a single test case by using many CPU cores. - * [afl-ddmin-mod](https://github.com/MarkusTeufelberger/afl-ddmin-mod) - a variation of afl-tmin based on the ddmin algorithm. - * [halfempty](https://github.com/googleprojectzero/halfempty) - is a fast utility for minimizing test cases by Tavis Ormandy based on parallelization. - -Distributed execution: - * [disfuzz-afl](https://github.com/MartijnB/disfuzz-afl) - distributed fuzzing for AFL. - * [AFLDFF](https://github.com/quantumvm/AFLDFF) - AFL distributed fuzzing framework. - * [afl-launch](https://github.com/bnagy/afl-launch) - a tool for the execution of many AFL instances. - * [afl-mothership](https://github.com/afl-mothership/afl-mothership) - management and execution of many synchronized AFL fuzzers on AWS cloud. - * [afl-in-the-cloud](https://github.com/abhisek/afl-in-the-cloud) - another script for running AFL in AWS. - -Deployment, management, monitoring, reporting - * [afl-utils](https://gitlab.com/rc0r/afl-utils) - a set of utilities for automatic processing/analysis of crashes and reducing the number of test cases. - * [afl-other-arch](https://github.com/shellphish/afl-other-arch) - is a set of patches and scripts for easily adding support for various non-x86 architectures for AFL. - * [afl-trivia](https://github.com/bnagy/afl-trivia) - a few small scripts to simplify the management of AFL. - * [afl-monitor](https://github.com/reflare/afl-monitor) - a script for monitoring AFL. - * [afl-manager](https://github.com/zx1340/afl-manager) - a web server on Python for managing multi-afl. - * [afl-remote](https://github.com/block8437/afl-remote) - a web server for the remote management of AFL instances. - * [afl-extras](https://github.com/fekir/afl-extras) - shell scripts to parallelize afl-tmin, startup, and data collection. - -Crash processing - * [afl-crash-analyzer](https://github.com/floyd-fuh/afl-crash-analyzer) - another crash analyzer for AFL. - * [fuzzer-utils](https://github.com/ThePatrickStar/fuzzer-utils) - a set of scripts for the analysis of results. - * [atriage](https://github.com/Ayrx/atriage) - a simple triage tool. - * [afl-kit](https://github.com/kcwu/afl-kit) - afl-cmin on Python. - * [AFLize](https://github.com/d33tah/aflize) - a tool that automatically generates builds of debian packages suitable for AFL. - * [afl-fid](https://github.com/FoRTE-Research/afl-fid) - a set of tools for working with input data. \ No newline at end of file diff --git a/docs/triaging_crashes.md b/docs/triaging_crashes.md deleted file mode 100644 index 21ccecaa..00000000 --- a/docs/triaging_crashes.md +++ /dev/null @@ -1,46 +0,0 @@ -# Triaging crashes - -The coverage-based grouping of crashes usually produces a small data set that -can be quickly triaged manually or with a very simple GDB or Valgrind script. -Every crash is also traceable to its parent non-crashing test case in the -queue, making it easier to diagnose faults. - -Having said that, it's important to acknowledge that some fuzzing crashes can be -difficult to quickly evaluate for exploitability without a lot of debugging and -code analysis work. To assist with this task, afl-fuzz supports a very unique -"crash exploration" mode enabled with the -C flag. - -In this mode, the fuzzer takes one or more crashing test cases as the input -and uses its feedback-driven fuzzing strategies to very quickly enumerate all -code paths that can be reached in the program while keeping it in the -crashing state. - -Mutations that do not result in a crash are rejected; so are any changes that -do not affect the execution path. - -The output is a small corpus of files that can be very rapidly examined to see -what degree of control the attacker has over the faulting address, or whether -it is possible to get past an initial out-of-bounds read - and see what lies -beneath. - -Oh, one more thing: for test case minimization, give afl-tmin a try. The tool -can be operated in a very simple way: - -```shell -./afl-tmin -i test_case -o minimized_result -- /path/to/program [...] -``` - -The tool works with crashing and non-crashing test cases alike. In the crash -mode, it will happily accept instrumented and non-instrumented binaries. In the -non-crashing mode, the minimizer relies on standard AFL++ instrumentation to make -the file simpler without altering the execution path. - -The minimizer accepts the -m, -t, -f and @@ syntax in a manner compatible with -afl-fuzz. - -Another tool in AFL++ is the afl-analyze tool. It takes an input -file, attempts to sequentially flip bytes, and observes the behavior of the -tested program. It then color-codes the input based on which sections appear to -be critical, and which are not; while not bulletproof, it can often offer quick -insights into complex file formats. More info about its operation can be found -near the end of [technical_details.md](technical_details.md). \ No newline at end of file diff --git a/docs/tutorials.md b/docs/tutorials.md index cc7ed130..ed8a7eec 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -1,6 +1,6 @@ # Tutorials -Here are some good writeups to show how to effectively use AFL++: +Here are some good write-ups to show how to effectively use AFL++: * [https://aflplus.plus/docs/tutorials/libxml2_tutorial/](https://aflplus.plus/docs/tutorials/libxml2_tutorial/) * [https://bananamafia.dev/post/gb-fuzz/](https://bananamafia.dev/post/gb-fuzz/) @@ -18,9 +18,13 @@ training, then we can highly recommend the following: If you are interested in fuzzing structured data (where you define what the structure is), these links have you covered: -* Superion for AFL++: [https://github.com/adrian-rt/superion-mutator](https://github.com/adrian-rt/superion-mutator) -* libprotobuf for AFL++: [https://github.com/P1umer/AFLplusplus-protobuf-mutator](https://github.com/P1umer/AFLplusplus-protobuf-mutator) -* libprotobuf raw: [https://github.com/bruce30262/libprotobuf-mutator_fuzzing_learning/tree/master/4_libprotobuf_aflpp_custom_mutator](https://github.com/bruce30262/libprotobuf-mutator_fuzzing_learning/tree/master/4_libprotobuf_aflpp_custom_mutator) -* libprotobuf for old AFL++ API: [https://github.com/thebabush/afl-libprotobuf-mutator](https://github.com/thebabush/afl-libprotobuf-mutator) +* Superion for AFL++: + [https://github.com/adrian-rt/superion-mutator](https://github.com/adrian-rt/superion-mutator) +* libprotobuf for AFL++: + [https://github.com/P1umer/AFLplusplus-protobuf-mutator](https://github.com/P1umer/AFLplusplus-protobuf-mutator) +* libprotobuf raw: + [https://github.com/bruce30262/libprotobuf-mutator_fuzzing_learning/tree/master/4_libprotobuf_aflpp_custom_mutator](https://github.com/bruce30262/libprotobuf-mutator_fuzzing_learning/tree/master/4_libprotobuf_aflpp_custom_mutator) +* libprotobuf for old AFL++ API: + [https://github.com/thebabush/afl-libprotobuf-mutator](https://github.com/thebabush/afl-libprotobuf-mutator) If you find other good ones, please send them to us :-) \ No newline at end of file |