From 5ec859cece70ab1b5cd9e0356c4cc3e260d2cbe0 Mon Sep 17 00:00:00 2001 From: llzmb <46303940+llzmb@users.noreply.github.com> Date: Sat, 20 Nov 2021 15:48:49 +0100 Subject: Clean up docs folder --- docs/fuzzing_in_depth.md | 630 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 630 insertions(+) create mode 100644 docs/fuzzing_in_depth.md (limited to 'docs/fuzzing_in_depth.md') diff --git a/docs/fuzzing_in_depth.md b/docs/fuzzing_in_depth.md new file mode 100644 index 00000000..5306cbef --- /dev/null +++ b/docs/fuzzing_in_depth.md @@ -0,0 +1,630 @@ +# Fuzzing with AFL++ + +The following describes how to fuzz with a target if source code is available. +If you have a binary-only target please skip to [#Instrumenting binary-only apps](#Instrumenting binary-only apps) + +Fuzzing source code is a three-step process. + +1. Compile the target with a special compiler that prepares the target to be + fuzzed efficiently. This step is called "instrumenting a target". +2. Prepare the fuzzing by selecting and optimizing the input corpus for the + target. +3. Perform the fuzzing of the target by randomly mutating input and assessing + if a generated input was processed in a new path in the target binary. + +### 1. Instrumenting that target + +#### a) Selecting the best AFL++ compiler for instrumenting the target + +AFL++ comes with a central compiler `afl-cc` that incorporates various different +kinds of compiler targets and and instrumentation options. +The following evaluation flow will help you to select the best possible. + +It is highly recommended to have the newest llvm version possible installed, +anything below 9 is not recommended. + +``` ++--------------------------------+ +| clang/clang++ 11+ is available | --> use LTO mode (afl-clang-lto/afl-clang-lto++) ++--------------------------------+ see [instrumentation/README.lto.md](instrumentation/README.lto.md) + | + | if not, or if the target fails with LTO afl-clang-lto/++ + | + v ++---------------------------------+ +| clang/clang++ 3.8+ is available | --> use LLVM mode (afl-clang-fast/afl-clang-fast++) ++---------------------------------+ see [instrumentation/README.llvm.md](instrumentation/README.llvm.md) + | + | if not, or if the target fails with LLVM afl-clang-fast/++ + | + v + +--------------------------------+ + | gcc 5+ is available | -> use GCC_PLUGIN mode (afl-gcc-fast/afl-g++-fast) + +--------------------------------+ see [instrumentation/README.gcc_plugin.md](instrumentation/README.gcc_plugin.md) and + [instrumentation/README.instrument_list.md](instrumentation/README.instrument_list.md) + | + | if not, or if you do not have a gcc with plugin support + | + v + use GCC mode (afl-gcc/afl-g++) (or afl-clang/afl-clang++ for clang) +``` + +Clickable README links for the chosen compiler: + + * [LTO mode - afl-clang-lto](../instrumentation/README.lto.md) + * [LLVM mode - afl-clang-fast](../instrumentation/README.llvm.md) + * [GCC_PLUGIN mode - afl-gcc-fast](../instrumentation/README.gcc_plugin.md) + * GCC/CLANG modes (afl-gcc/afl-clang) have no README as they have no own features + +You can select the mode for the afl-cc compiler by: + 1. use a symlink to afl-cc: afl-gcc, afl-g++, afl-clang, afl-clang++, + afl-clang-fast, afl-clang-fast++, afl-clang-lto, afl-clang-lto++, + afl-gcc-fast, afl-g++-fast (recommended!) + 2. using the environment variable AFL_CC_COMPILER with MODE + 3. passing --afl-MODE command line options to the compiler via CFLAGS/CXXFLAGS/CPPFLAGS + +MODE can be one of: LTO (afl-clang-lto*), LLVM (afl-clang-fast*), GCC_PLUGIN +(afl-g*-fast) or GCC (afl-gcc/afl-g++) or CLANG(afl-clang/afl-clang++). + +Because no AFL specific command-line options are accepted (beside the +--afl-MODE command), the compile-time tools make fairly broad use of environment +variables, which can be listed with `afl-cc -hh` or by reading [env_variables.md](env_variables.md). + +#### b) Selecting instrumentation options + +The following options are available when you instrument with LTO mode (afl-clang-fast/afl-clang-lto): + + * Splitting integer, string, float and switch comparisons so AFL++ can easier + solve these. This is an important option if you do not have a very good + and large input corpus. This technique is called laf-intel or COMPCOV. + To use this set the following environment variable before compiling the + target: `export AFL_LLVM_LAF_ALL=1` + You can read more about this in [instrumentation/README.laf-intel.md](../instrumentation/README.laf-intel.md) + * A different technique (and usually a better one than laf-intel) is to + instrument the target so that any compare values in the target are sent to + AFL++ which then tries to put these values into the fuzzing data at different + locations. This technique is very fast and good - if the target does not + transform input data before comparison. Therefore this technique is called + `input to state` or `redqueen`. + If you want to use this technique, then you have to compile the target + twice, once specifically with/for this mode by setting `AFL_LLVM_CMPLOG=1`, + and pass this binary to afl-fuzz via the `-c` parameter. + Note that you can compile also just a cmplog binary and use that for both + however there will be a performance penality. + You can read more about this in [instrumentation/README.cmplog.md](../instrumentation/README.cmplog.md) + +If you use LTO, LLVM or GCC_PLUGIN mode (afl-clang-fast/afl-clang-lto/afl-gcc-fast) +you have the option to selectively only instrument parts of the target that you +are interested in: + + * To instrument only those parts of the target that you are interested in + create a file with all the filenames of the source code that should be + instrumented. + For afl-clang-lto and afl-gcc-fast - or afl-clang-fast if a mode other than + DEFAULT/PCGUARD is used or you have llvm > 10.0.0 - just put one + filename or function per line (no directory information necessary for + filenames9, and either set `export AFL_LLVM_ALLOWLIST=allowlist.txt` **or** + `export AFL_LLVM_DENYLIST=denylist.txt` - depending on if you want per + default to instrument unless noted (DENYLIST) or not perform instrumentation + unless requested (ALLOWLIST). + **NOTE:** During optimization functions might be inlined and then would not match! + See [instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md) + +There are many more options and modes available however these are most of the +time less effective. See: + * [instrumentation/README.ctx.md](../instrumentation/README.ctx.md) + * [instrumentation/README.ngram.md](../instrumentation/README.ngram.md) + +AFL++ performs "never zero" counting in its bitmap. You can read more about this +here: + * [instrumentation/README.neverzero.md](../instrumentation/README.neverzero.md) + +#### c) Sanitizers + +It is possible to use sanitizers when instrumenting targets for fuzzing, +which allows you to find bugs that would not necessarily result in a crash. + +Note that sanitizers have a huge impact on CPU (= less executions per second) +and RAM usage. Also you should only run one afl-fuzz instance per sanitizer type. +This is enough because a use-after-free bug will be picked up, e.g. by +ASAN (address sanitizer) anyway when syncing to other fuzzing instances, +so not all fuzzing instances need to be instrumented with ASAN. + +The following sanitizers have built-in support in AFL++: + * ASAN = Address SANitizer, finds memory corruption vulnerabilities like + use-after-free, NULL pointer dereference, buffer overruns, etc. + Enabled with `export AFL_USE_ASAN=1` before compiling. + * MSAN = Memory SANitizer, finds read access to uninitialized memory, eg. + a local variable that is defined and read before it is even set. + Enabled with `export AFL_USE_MSAN=1` before compiling. + * UBSAN = Undefined Behaviour SANitizer, finds instances where - by the + C and C++ standards - undefined behaviour happens, e.g. adding two + signed integers together where the result is larger than a signed integer + can hold. + Enabled with `export AFL_USE_UBSAN=1` before compiling. + * CFISAN = Control Flow Integrity SANitizer, finds instances where the + control flow is found to be illegal. Originally this was rather to + prevent return oriented programming exploit chains from functioning, + in fuzzing this is mostly reduced to detecting type confusion + vulnerabilities - which is however one of the most important and dangerous + C++ memory corruption classes! + Enabled with `export AFL_USE_CFISAN=1` before compiling. + * TSAN = Thread SANitizer, finds thread race conditions. + Enabled with `export AFL_USE_TSAN=1` before compiling. + * LSAN = Leak SANitizer, finds memory leaks in a program. This is not really + a security issue, but for developers this can be very valuable. + Note that unlike the other sanitizers above this needs + `__AFL_LEAK_CHECK();` added to all areas of the target source code where you + find a leak check necessary! + Enabled with `export AFL_USE_LSAN=1` before compiling. + +It is possible to further modify the behaviour of the sanitizers at run-time +by setting `ASAN_OPTIONS=...`, `LSAN_OPTIONS` etc. - the available parameters +can be looked up in the sanitizer documentation of llvm/clang. +afl-fuzz however requires some specific parameters important for fuzzing to be +set. If you want to set your own, it might bail and report what it is missing. + +Note that some sanitizers cannot be used together, e.g. ASAN and MSAN, and +others often cannot work together because of target weirdness, e.g. ASAN and +CFISAN. You might need to experiment which sanitizers you can combine in a +target (which means more instances can be run without a sanitized target, +which is more effective). + +#### d) Modify the target + +If the target has features that make fuzzing more difficult, e.g. +checksums, HMAC, etc. then modify the source code so that checks for these +values are removed. +This can even be done safely for source code used in operational products +by eliminating these checks within these AFL specific blocks: + +``` +#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION + // say that the checksum or HMAC was fine - or whatever is required + // to eliminate the need for the fuzzer to guess the right checksum + return 0; +#endif +``` + +All AFL++ compilers will set this preprocessor definition automatically. + +#### e) Instrument the target + +In this step the target source code is compiled so that it can be fuzzed. + +Basically you have to tell the target build system that the selected AFL++ +compiler is used. Also - if possible - you should always configure the +build system such that the target is compiled statically and not dynamically. +How to do this is described below. + +The #1 rule when instrumenting a target is: avoid instrumenting shared +libraries at all cost. You would need to set LD_LIBRARY_PATH to point to +these, you could accidently type "make install" and install them system wide - +so don't. Really don't. +**Always compile libraries you want to have instrumented as static and link +these to the target program!** + +Then build the target. (Usually with `make`) + +**NOTES** + +1. sometimes configure and build systems are fickle and do not like + stderr output (and think this means a test failure) - which is something + AFL++ likes to do to show statistics. It is recommended to disable AFL++ + instrumentation reporting via `export AFL_QUIET=1`. + +2. sometimes configure and build systems error on warnings - these should be + disabled (e.g. `--disable-werror` for some configure scripts). + +3. in case the configure/build system complains about AFL++'s compiler and + aborts then set `export AFL_NOOPT=1` which will then just behave like the + real compiler. This option has to be unset again before building the target! + +##### configure + +For `configure` build systems this is usually done by: +`CC=afl-clang-fast CXX=afl-clang-fast++ ./configure --disable-shared` + +Note that if you are using the (better) afl-clang-lto compiler you also have to +set AR to llvm-ar[-VERSION] and RANLIB to llvm-ranlib[-VERSION] - as is +described in [instrumentation/README.lto.md](../instrumentation/README.lto.md). + +##### cmake + +For `cmake` build systems this is usually done by: +`mkdir build; cd build; cmake -DCMAKE_C_COMPILER=afl-cc -DCMAKE_CXX_COMPILER=afl-c++ ..` + +Note that if you are using the (better) afl-clang-lto compiler you also have to +set AR to llvm-ar[-VERSION] and RANLIB to llvm-ranlib[-VERSION] - as is +described in [instrumentation/README.lto.md](../instrumentation/README.lto.md). + +##### meson + +For meson you have to set the AFL++ compiler with the very first command! +`CC=afl-cc CXX=afl-c++ meson` + +##### other build systems or if configure/cmake didn't work + +Sometimes cmake and configure do not pick up the AFL++ compiler, or the +ranlib/ar that is needed - because this was just not foreseen by the developer +of the target. Or they have non-standard options. Figure out if there is a +non-standard way to set this, otherwise set up the build normally and edit the +generated build environment afterwards manually to point it to the right compiler +(and/or ranlib and ar). + +#### f) Better instrumentation + +If you just fuzz a target program as-is you are wasting a great opportunity for +much more fuzzing speed. + +This variant requires the usage of afl-clang-lto, afl-clang-fast or afl-gcc-fast. + +It is the so-called `persistent mode`, which is much, much faster but +requires that you code a source file that is specifically calling the target +functions that you want to fuzz, plus a few specific AFL++ functions around +it. See [instrumentation/README.persistent_mode.md](../instrumentation/README.persistent_mode.md) for details. + +Basically if you do not fuzz a target in persistent mode then you are just +doing it for a hobby and not professionally :-). + +#### g) libfuzzer fuzzer harnesses with LLVMFuzzerTestOneInput() + +libfuzzer `LLVMFuzzerTestOneInput()` harnesses are the defacto standard +for fuzzing, and they can be used with AFL++ (and honggfuzz) as well! +Compiling them is as simple as: +``` +afl-clang-fast++ -fsanitize=fuzzer -o harness harness.cpp targetlib.a +``` +You can even use advanced libfuzzer features like `FuzzedDataProvider`, +`LLVMFuzzerMutate()` etc. and they will work! + +The generated binary is fuzzed with afl-fuzz like any other fuzz target. + +Bonus: the target is already optimized for fuzzing due to persistent mode and +shared-memory testcases and hence gives you the fastest speed possible. + +For more information see [utils/aflpp_driver/README.md](../utils/aflpp_driver/README.md) + +### 2. Preparing the fuzzing campaign + +As you fuzz the target with mutated input, having as diverse inputs for the +target as possible improves the efficiency a lot. + +#### a) Collect inputs + +Try to gather valid inputs for the target from wherever you can. E.g. if it is +the PNG picture format try to find as many png files as possible, e.g. from +reported bugs, test suites, random downloads from the internet, unit test +case data - from all kind of PNG software. + +If the input format is not known, you can also modify a target program to write +normal data it receives and processes to a file and use these. + +#### b) Making the input corpus unique + +Use the AFL++ tool `afl-cmin` to remove inputs from the corpus that do not +produce a new path in the target. + +Put all files from step a) into one directory, e.g. INPUTS. + +If the target program is to be called by fuzzing as `bin/target -d INPUTFILE` +the run afl-cmin like this: +`afl-cmin -i INPUTS -o INPUTS_UNIQUE -- bin/target -d @@` +Note that the INPUTFILE argument that the target program would read from has to be set as `@@`. + +If the target reads from stdin instead, just omit the `@@` as this is the +default. + +This step is highly recommended! + +#### c) Minimizing all corpus files + +The shorter the input files that still traverse the same path +within the target, the better the fuzzing will be. This minimization +is done with `afl-tmin` however it is a long process as this has to +be done for every file: + +``` +mkdir input +cd INPUTS_UNIQUE +for i in *; do + afl-tmin -i "$i" -o "../input/$i" -- bin/target -d @@ +done +``` + +This step can also be parallelized, e.g. with `parallel`. +Note that this step is rather optional though. + +#### Done! + +The INPUTS_UNIQUE/ directory from step b) - or even better the directory input/ +if you minimized the corpus in step c) - is the resulting input corpus directory +to be used in fuzzing! :-) + +### 3. Fuzzing the target + +In this final step we fuzz the target. +There are not that many important options to run the target - unless you want +to use many CPU cores/threads for the fuzzing, which will make the fuzzing much +more useful. + +If you just use one CPU for fuzzing, then you are fuzzing just for fun and not +seriously :-) + +#### a) Running afl-fuzz + +Before you do even a test run of afl-fuzz execute `sudo afl-system-config` (on +the host if you execute afl-fuzz in a docker container). This reconfigures the +system for optimal speed - which afl-fuzz checks and bails otherwise. +Set `export AFL_SKIP_CPUFREQ=1` for afl-fuzz to skip this check if you cannot +run afl-system-config with root privileges on the host for whatever reason. + +Note there is also `sudo afl-persistent-config` which sets additional permanent +boot options for a much better fuzzing performance. + +Note that both scripts improve your fuzzing performance but also decrease your +system protection against attacks! So set strong firewall rules and only +expose SSH as a network service if you use these (which is highly recommended). + +If you have an input corpus from step 2 then specify this directory with the `-i` +option. Otherwise create a new directory and create a file with any content +as test data in there. + +If you do not want anything special, the defaults are already usually best, +hence all you need is to specify the seed input directory with the result of +step [2a. Collect inputs](#a-collect-inputs): +`afl-fuzz -i input -o output -- bin/target -d @@` +Note that the directory specified with -o will be created if it does not exist. + +It can be valuable to run afl-fuzz in a screen or tmux shell so you can log off, +or afl-fuzz is not aborted if you are running it in a remote ssh session where +the connection fails in between. +Only do that though once you have verified that your fuzzing setup works! +Simply run it like `screen -dmS afl-main -- afl-fuzz -M main-$HOSTNAME -i ...` +and it will start away in a screen session. To enter this session simply type +`screen -r afl-main`. You see - it makes sense to name the screen session +same as the afl-fuzz -M/-S naming :-) +For more information on screen or tmux please check their documentation. + +If you need to stop and re-start the fuzzing, use the same command line options +(or even change them by selecting a different power schedule or another +mutation mode!) and switch the input directory with a dash (`-`): +`afl-fuzz -i - -o output -- bin/target -d @@` + +Memory limits are not enforced by afl-fuzz by default and the system may run +out of memory. You can decrease the memory with the `-m` option, the value is +in MB. If this is too small for the target, you can usually see this by +afl-fuzz bailing with the message that it could not connect to the forkserver. + +Adding a dictionary is helpful. See the directory [dictionaries/](../dictionaries/) if +something is already included for your data format, and tell afl-fuzz to load +that dictionary by adding `-x dictionaries/FORMAT.dict`. With afl-clang-lto +you have an autodictionary generation for which you need to do nothing except +to use afl-clang-lto as the compiler. You also have the option to generate +a dictionary yourself, see [utils/libtokencap/README.md](../utils/libtokencap/README.md). + +afl-fuzz has a variety of options that help to workaround target quirks like +specific locations for the input file (`-f`), performing deterministic +fuzzing (`-D`) and many more. Check out `afl-fuzz -h`. + +We highly recommend that you set a memory limit for running the target with `-m` +which defines the maximum memory in MB. This prevents a potential +out-of-memory problem for your system plus helps you detect missing `malloc()` +failure handling in the target. +Play around with various -m values until you find one that safely works for all +your input seeds (if you have good ones and then double or quadrouple that. + +By default afl-fuzz never stops fuzzing. To terminate AFL++ simply press Control-C +or send a signal SIGINT. You can limit the number of executions or approximate runtime +in seconds with options also. + +When you start afl-fuzz you will see a user interface that shows what the status +is: +![resources/screenshot.png](resources/screenshot.png) + +All labels are explained in [status_screen.md](status_screen.md). + +#### b) Using multiple cores + +If you want to seriously fuzz then use as many cores/threads as possible to +fuzz your target. + +On the same machine - due to the design of how AFL++ works - there is a maximum +number of CPU cores/threads that are useful, use more and the overall performance +degrades instead. This value depends on the target, and the limit is between 32 +and 64 cores per machine. + +If you have the RAM, it is highly recommended run the instances with a caching +of the testcases. Depending on the average testcase size (and those found +during fuzzing) and their number, a value between 50-500MB is recommended. +You can set the cache size (in MB) by setting the environment variable `AFL_TESTCACHE_SIZE`. + +There should be one main fuzzer (`-M main-$HOSTNAME` option) and as many secondary +fuzzers (eg `-S variant1`) as you have cores that you use. +Every -M/-S entry needs a unique name (that can be whatever), however the same +-o output directory location has to be used for all instances. + +For every secondary fuzzer there should be a variation, e.g.: + * one should fuzz the target that was compiled differently: with sanitizers + activated (`export AFL_USE_ASAN=1 ; export AFL_USE_UBSAN=1 ; + export AFL_USE_CFISAN=1`) + * one or two should fuzz the target with CMPLOG/redqueen (see above), at + least one cmplog instance should follow transformations (`-l AT`) + * one to three fuzzers should fuzz a target compiled with laf-intel/COMPCOV + (see above). Important note: If you run more than one laf-intel/COMPCOV + fuzzer and you want them to share their intermediate results, the main + fuzzer (`-M`) must be one of the them! (Although this is not really + recommended.) + +All other secondaries should be used like this: + * A quarter to a third with the MOpt mutator enabled: `-L 0` + * run with a different power schedule, recommended are: + `fast (default), explore, coe, lin, quad, exploit and rare` + which you can set with e.g. `-p explore` + * a few instances should use the old queue cycling with `-Z` + +Also it is recommended to set `export AFL_IMPORT_FIRST=1` to load testcases +from other fuzzers in the campaign first. + +If you have a large corpus, a corpus from a previous run or are fuzzing in +a CI, then also set `export AFL_CMPLOG_ONLY_NEW=1` and `export AFL_FAST_CAL=1`. + +You can also use different fuzzers. +If you are using AFL spinoffs or AFL conforming fuzzers, then just use the +same -o directory and give it a unique `-S` name. +Examples are: + * [Fuzzolic](https://github.com/season-lab/fuzzolic) + * [symcc](https://github.com/eurecom-s3/symcc/) + * [Eclipser](https://github.com/SoftSec-KAIST/Eclipser/) + * [AFLsmart](https://github.com/aflsmart/aflsmart) + * [FairFuzz](https://github.com/carolemieux/afl-rb) + * [Neuzz](https://github.com/Dongdongshe/neuzz) + * [Angora](https://github.com/AngoraFuzzer/Angora) + +A long list can be found at [https://github.com/Microsvuln/Awesome-AFL](https://github.com/Microsvuln/Awesome-AFL) + +However you can also sync AFL++ with honggfuzz, libfuzzer with `-entropic=1`, etc. +Just show the main fuzzer (-M) with the `-F` option where the queue/work +directory of a different fuzzer is, e.g. `-F /src/target/honggfuzz`. +Using honggfuzz (with `-n 1` or `-n 2`) and libfuzzer in parallel is highly +recommended! + +#### c) Using multiple machines for fuzzing + +Maybe you have more than one machine you want to fuzz the same target on. +Simply start the `afl-fuzz` (and perhaps libfuzzer, honggfuzz, ...) +orchestra as you like, just ensure that your have one and only one `-M` +instance per server, and that its name is unique, hence the recommendation +for `-M main-$HOSTNAME`. + +Now there are three strategies on how you can sync between the servers: + * never: sounds weird, but this makes every server an island and has the + chance the each follow different paths into the target. You can make + this even more interesting by even giving different seeds to each server. + * regularly (~4h): this ensures that all fuzzing campaigns on the servers + "see" the same thing. It is like fuzzing on a huge server. + * in intervals of 1/10th of the overall expected runtime of the fuzzing you + sync. This tries a bit to combine both. have some individuality of the + paths each campaign on a server explores, on the other hand if one + gets stuck where another found progress this is handed over making it + unstuck. + +The syncing process itself is very simple. +As the `-M main-$HOSTNAME` instance syncs to all `-S` secondaries as well +as to other fuzzers, you have to copy only this directory to the other +machines. + +Lets say all servers have the `-o out` directory in /target/foo/out, and +you created a file `servers.txt` which contains the hostnames of all +participating servers, plus you have an ssh key deployed to all of them, +then run: +```bash +for FROM in `cat servers.txt`; do + for TO in `cat servers.txt`; do + rsync -rlpogtz --rsh=ssh $FROM:/target/foo/out/main-$FROM $TO:target/foo/out/ + done +done +``` +You can run this manually, per cron job - as you need it. +There is a more complex and configurable script in `utils/distributed_fuzzing`. + +#### d) The status of the fuzz campaign + +AFL++ comes with the `afl-whatsup` script to show the status of the fuzzing +campaign. + +Just supply the directory that afl-fuzz is given with the -o option and +you will see a detailed status of every fuzzer in that campaign plus +a summary. + +To have only the summary use the `-s` switch e.g.: `afl-whatsup -s out/` + +If you have multiple servers then use the command after a sync, or you have +to execute this script per server. + +Another tool to inspect the current state and history of a specific instance +is afl-plot, which generates an index.html file and a graphs that show how +the fuzzing instance is performing. +The syntax is `afl-plot instance_dir web_dir`, e.g. `afl-plot out/default /srv/www/htdocs/plot` + +#### e) Stopping fuzzing, restarting fuzzing, adding new seeds + +To stop an afl-fuzz run, simply press Control-C. + +To restart an afl-fuzz run, just reuse the same command line but replace the +`-i directory` with `-i -` or set `AFL_AUTORESUME=1`. + +If you want to add new seeds to a fuzzing campaign you can run a temporary +fuzzing instance, e.g. when your main fuzzer is using `-o out` and the new +seeds are in `newseeds/` directory: +``` +AFL_BENCH_JUST_ONE=1 AFL_FAST_CAL=1 afl-fuzz -i newseeds -o out -S newseeds -- ./target +``` + +#### f) Checking the coverage of the fuzzing + +The `paths found` value is a bad indicator for checking how good the coverage is. + +A better indicator - if you use default llvm instrumentation with at least +version 9 - is to use `afl-showmap` with the collect coverage option `-C` on +the output directory: +``` +$ afl-showmap -C -i out -o /dev/null -- ./target -params @@ +... +[*] Using SHARED MEMORY FUZZING feature. +[*] Target map size: 9960 +[+] Processed 7849 input files. +[+] Captured 4331 tuples (highest value 255, total values 67130596) in '/dev/nul +l'. +[+] A coverage of 4331 edges were achieved out of 9960 existing (43.48%) with 7849 input files. +``` +It is even better to check out the exact lines of code that have been reached - +and which have not been found so far. + +An "easy" helper script for this is [https://github.com/vanhauser-thc/afl-cov](https://github.com/vanhauser-thc/afl-cov), +just follow the README of that separate project. + +If you see that an important area or a feature has not been covered so far then +try to find an input that is able to reach that and start a new secondary in +that fuzzing campaign with that seed as input, let it run for a few minutes, +then terminate it. The main node will pick it up and make it available to the +other secondary nodes over time. Set `export AFL_NO_AFFINITY=1` or +`export AFL_TRY_AFFINITY=1` if you have no free core. + +Note that in nearly all cases you can never reach full coverage. A lot of +functionality is usually dependent on exclusive options that would need individual +fuzzing campaigns each with one of these options set. E.g. if you fuzz a library to +convert image formats and your target is the png to tiff API then you will not +touch any of the other library APIs and features. + +#### g) How long to fuzz a target? + +This is a difficult question. +Basically if no new path is found for a long time (e.g. for a day or a week) +then you can expect that your fuzzing won't be fruitful anymore. +However often this just means that you should switch out secondaries for +others, e.g. custom mutator modules, sync to very different fuzzers, etc. + +Keep the queue/ directory (for future fuzzings of the same or similar targets) +and use them to seed other good fuzzers like libfuzzer with the -entropic +switch or honggfuzz. + +#### h) Improve the speed! + + * Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 speed increase) + * If you do not use shmem persistent mode, use `AFL_TMPDIR` to point the input file on a tempfs location, see [env_variables.md](env_variables.md) + * Linux: Improve kernel performance: modify `/etc/default/grub`, set `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then `update-grub` and `reboot` (warning: makes the system more insecure) - you can also just run `sudo afl-persistent-config` + * Linux: Running on an `ext2` filesystem with `noatime` mount option will be a bit faster than on any other journaling filesystem + * Use your cores! [b) Using multiple cores](#b-using-multiple-cores) + * Run `sudo afl-system-config` before starting the first afl-fuzz instance after a reboot + +### The End + +Check out the [FAQ](FAQ.md) if it maybe answers your question (that +you might not even have known you had ;-) ). + +This is basically all you need to know to professionally run fuzzing campaigns. +If you want to know more, the tons of texts in [docs/](./) will have you covered. + +Note that there are also a lot of tools out there that help fuzzing with AFL++ +(some might be deprecated or unsupported), see [third_party_tools.md](third_party_tools.md). \ No newline at end of file -- cgit 1.4.1 From c31f4646cbd00f591dad3258c08ff8e56aa94420 Mon Sep 17 00:00:00 2001 From: llzmb <46303940+llzmb@users.noreply.github.com> Date: Sun, 21 Nov 2021 21:11:52 +0100 Subject: Clean up docs folder --- README.md | 29 +- docs/afl-fuzz_approach.md | 540 ++++++++++++++++++++++++++- docs/custom_mutators.md | 6 +- docs/env_variables.md | 9 +- docs/features.md | 2 +- docs/fuzzing_in_depth.md | 572 +++++++++++++++-------------- docs/important_changes.md | 4 +- docs/interpreting_output.md | 71 ---- docs/status_screen.md | 444 ---------------------- docs/third_party_tools.md | 6 +- qemu_mode/libqasan/README.md | 2 +- unicorn_mode/samples/persistent/COMPILE.md | 12 +- utils/aflpp_driver/README.md | 12 +- 13 files changed, 864 insertions(+), 845 deletions(-) delete mode 100644 docs/interpreting_output.md delete mode 100644 docs/status_screen.md (limited to 'docs/fuzzing_in_depth.md') diff --git a/README.md b/README.md index fcb6b3c9..e0cb4558 100644 --- a/README.md +++ b/README.md @@ -80,8 +80,10 @@ Step-by-step quick start: 1. Compile the program or library to be fuzzed using `afl-cc`. A common way to do this would be: - CC=/path/to/afl-cc CXX=/path/to/afl-c++ ./configure --disable-shared - make clean all + ``` + CC=/path/to/afl-cc CXX=/path/to/afl-c++ ./configure --disable-shared + make clean all + ``` 2. Get a small but valid input file that makes sense to the program. When fuzzing verbose syntax (SQL, HTTP, etc), create a dictionary as described in @@ -89,10 +91,10 @@ Step-by-step quick start: 3. If the program reads from stdin, run `afl-fuzz` like so: -``` + ``` ./afl-fuzz -i seeds_dir -o output_dir -- \ - /path/to/tested/program [...program's cmdline...] -``` + /path/to/tested/program [...program's cmdline...] + ``` To add a dictionary, add `-x /path/to/dictionary.txt` to afl-fuzz. @@ -100,13 +102,20 @@ Step-by-step quick start: command line; AFL will put an auto-generated file name in there for you. 4. Investigate anything shown in red in the fuzzer UI by promptly consulting - [docs/status_screen.md](docs/status_screen.md). + [docs/afl-fuzz_approach.md#understanding-the-status-screen](docs/afl-fuzz_approach.md#understanding-the-status-screen). + +5. Interpret the output, see + [docs/afl-fuzz_approach.md#interpreting-output](docs/afl-fuzz_approach.md#interpreting-output). -5. You will find found crashes and hangs in the subdirectories `crashes/` and +6. You will find found crashes and hangs in the subdirectories `crashes/` and `hangs/` in the `-o output_dir` directory. You can replay the crashes by - feeding them to the target, e.g.: `cat output_dir/crashes/id:000000,* | - /path/to/tested/program [...program's cmdline...]` You can generate cores or - use gdb directly to follow up the crashes. + feeding them to the target, e.g.: + + ``` + cat output_dir/crashes/id:000000,* | /path/to/tested/program [...program's cmdline...] + ``` + + You can generate cores or use gdb directly to follow up the crashes. ## Contact diff --git a/docs/afl-fuzz_approach.md b/docs/afl-fuzz_approach.md index 5652816b..57a275d9 100644 --- a/docs/afl-fuzz_approach.md +++ b/docs/afl-fuzz_approach.md @@ -1,37 +1,539 @@ # The afl-fuzz approach -American Fuzzy Lop is a brute-force fuzzer coupled with an exceedingly simple -but rock-solid instrumentation-guided genetic algorithm. It uses a modified -form of edge coverage to effortlessly pick up subtle, local-scale changes to -program control flow. +AFL++ is a brute-force fuzzer coupled with an exceedingly simple but rock-solid +instrumentation-guided genetic algorithm. It uses a modified form of edge +coverage to effortlessly pick up subtle, local-scale changes to program control +flow. Simplifying a bit, the overall algorithm can be summed up as: - 1) Load user-supplied initial test cases into the queue, +1) Load user-supplied initial test cases into the queue. - 2) Take the next input file from the queue, +2) Take the next input file from the queue. - 3) Attempt to trim the test case to the smallest size that doesn't alter - the measured behavior of the program, +3) Attempt to trim the test case to the smallest size that doesn't alter the + measured behavior of the program. - 4) Repeatedly mutate the file using a balanced and well-researched variety - of traditional fuzzing strategies, +4) Repeatedly mutate the file using a balanced and well-researched variety of + traditional fuzzing strategies. - 5) If any of the generated mutations resulted in a new state transition - recorded by the instrumentation, add mutated output as a new entry in the - queue. +5) If any of the generated mutations resulted in a new state transition recorded + by the instrumentation, add mutated output as a new entry in the queue. - 6) Go to 2. +6) Go to 2. The discovered test cases are also periodically culled to eliminate ones that have been obsoleted by newer, higher-coverage finds; and undergo several other instrumentation-driven effort minimization steps. As a side result of the fuzzing process, the tool creates a small, -self-contained corpus of interesting test cases. These are extremely useful -for seeding other, labor- or resource-intensive testing regimes - for example, -for stress-testing browsers, office applications, graphics suites, or -closed-source tools. +self-contained corpus of interesting test cases. These are extremely useful for +seeding other, labor- or resource-intensive testing regimes - for example, for +stress-testing browsers, office applications, graphics suites, or closed-source +tools. The fuzzer is thoroughly tested to deliver out-of-the-box performance far -superior to blind fuzzing or coverage-only tools. \ No newline at end of file +superior to blind fuzzing or coverage-only tools. + +## Understanding the status screen + +This document provides an overview of the status screen - plus tips for +troubleshooting any warnings and red text shown in the UI. See +[README.md](../README.md) for the general instruction manual. + +### A note about colors + +The status screen and error messages use colors to keep things readable and +attract your attention to the most important details. For example, red almost +always means "consult this doc" :-) + +Unfortunately, the UI will render correctly only if your terminal is using +traditional un*x palette (white text on black background) or something close to +that. + +If you are using inverse video, you may want to change your settings, say: + +- For GNOME Terminal, go to `Edit > Profile` preferences, select the "colors" + tab, and from the list of built-in schemes, choose "white on black". +- For the MacOS X Terminal app, open a new window using the "Pro" scheme via the + `Shell > New Window` menu (or make "Pro" your default). + +Alternatively, if you really like your current colors, you can edit config.h to +comment out USE_COLORS, then do `make clean all`. + +I'm not aware of any other simple way to make this work without causing other +side effects - sorry about that. + +With that out of the way, let's talk about what's actually on the screen... + +### The status bar + +``` +american fuzzy lop ++3.01a (default) [fast] {0} +``` + +The top line shows you which mode afl-fuzz is running in (normal: "american +fuzzy lop", crash exploration mode: "peruvian rabbit mode") and the version of +AFL++. Next to the version is the banner, which, if not set with -T by hand, +will either show the binary name being fuzzed, or the -M/-S main/secondary name +for parallel fuzzing. Second to last is the power schedule mode being run +(default: fast). Finally, the last item is the CPU id. + +### Process timing + +``` + +----------------------------------------------------+ + | run time : 0 days, 8 hrs, 32 min, 43 sec | + | last new path : 0 days, 0 hrs, 6 min, 40 sec | + | last uniq crash : none seen yet | + | last uniq hang : 0 days, 1 hrs, 24 min, 32 sec | + +----------------------------------------------------+ +``` + +This section is fairly self-explanatory: it tells you how long the fuzzer has +been running and how much time has elapsed since its most recent finds. This is +broken down into "paths" (a shorthand for test cases that trigger new execution +patterns), crashes, and hangs. + +When it comes to timing: there is no hard rule, but most fuzzing jobs should be +expected to run for days or weeks; in fact, for a moderately complex project, +the first pass will probably take a day or so. Every now and then, some jobs +will be allowed to run for months. + +There's one important thing to watch out for: if the tool is not finding new +paths within several minutes of starting, you're probably not invoking the +target binary correctly and it never gets to parse the input files we're +throwing at it; another possible explanations are that the default memory limit +(`-m`) is too restrictive, and the program exits after failing to allocate a +buffer very early on; or that the input files are patently invalid and always +fail a basic header check. + +If there are no new paths showing up for a while, you will eventually see a big +red warning in this section, too :-) + +### Overall results + +``` + +-----------------------+ + | cycles done : 0 | + | total paths : 2095 | + | uniq crashes : 0 | + | uniq hangs : 19 | + +-----------------------+ +``` + +The first field in this section gives you the count of queue passes done so far +- that is, the number of times the fuzzer went over all the interesting test +cases discovered so far, fuzzed them, and looped back to the very beginning. +Every fuzzing session should be allowed to complete at least one cycle; and +ideally, should run much longer than that. + +As noted earlier, the first pass can take a day or longer, so sit back and +relax. + +To help make the call on when to hit `Ctrl-C`, the cycle counter is color-coded. +It is shown in magenta during the first pass, progresses to yellow if new finds +are still being made in subsequent rounds, then blue when that ends - and +finally, turns green after the fuzzer hasn't been seeing any action for a longer +while. + +The remaining fields in this part of the screen should be pretty obvious: +there's the number of test cases ("paths") discovered so far, and the number of +unique faults. The test cases, crashes, and hangs can be explored in real-time +by browsing the output directory, as discussed in [README.md](../README.md). + +### Cycle progress + +``` + +-------------------------------------+ + | now processing : 1296 (61.86%) | + | paths timed out : 0 (0.00%) | + +-------------------------------------+ +``` + +This box tells you how far along the fuzzer is with the current queue cycle: it +shows the ID of the test case it is currently working on, plus the number of +inputs it decided to ditch because they were persistently timing out. + +The "*" suffix sometimes shown in the first line means that the currently +processed path is not "favored" (a property discussed later on). + +### Map coverage + +``` + +--------------------------------------+ + | map density : 10.15% / 29.07% | + | count coverage : 4.03 bits/tuple | + +--------------------------------------+ +``` + +The section provides some trivia about the coverage observed by the +instrumentation embedded in the target binary. + +The first line in the box tells you how many branch tuples we have already hit, +in proportion to how much the bitmap can hold. The number on the left describes +the current input; the one on the right is the value for the entire input +corpus. + +Be wary of extremes: + +- Absolute numbers below 200 or so suggest one of three things: that the program + is extremely simple; that it is not instrumented properly (e.g., due to being + linked against a non-instrumented copy of the target library); or that it is + bailing out prematurely on your input test cases. The fuzzer will try to mark + this in pink, just to make you aware. +- Percentages over 70% may very rarely happen with very complex programs that + make heavy use of template-generated code. Because high bitmap density makes + it harder for the fuzzer to reliably discern new program states, we recommend + recompiling the binary with `AFL_INST_RATIO=10` or so and trying again (see + [env_variables.md](env_variables.md)). The fuzzer will flag high percentages + in red. Chances are, you will never see that unless you're fuzzing extremely + hairy software (say, v8, perl, ffmpeg). + +The other line deals with the variability in tuple hit counts seen in the +binary. In essence, if every taken branch is always taken a fixed number of +times for all the inputs we have tried, this will read `1.00`. As we manage to +trigger other hit counts for every branch, the needle will start to move toward +`8.00` (every bit in the 8-bit map hit), but will probably never reach that +extreme. + +Together, the values can be useful for comparing the coverage of several +different fuzzing jobs that rely on the same instrumented binary. + +### Stage progress + +``` + +-------------------------------------+ + | now trying : interest 32/8 | + | stage execs : 3996/34.4k (11.62%) | + | total execs : 27.4M | + | exec speed : 891.7/sec | + +-------------------------------------+ +``` + +This part gives you an in-depth peek at what the fuzzer is actually doing right +now. It tells you about the current stage, which can be any of: + +- calibration - a pre-fuzzing stage where the execution path is examined to + detect anomalies, establish baseline execution speed, and so on. Executed very + briefly whenever a new find is being made. +- trim L/S - another pre-fuzzing stage where the test case is trimmed to the + shortest form that still produces the same execution path. The length (L) and + stepover (S) are chosen in general relationship to file size. +- bitflip L/S - deterministic bit flips. There are L bits toggled at any given + time, walking the input file with S-bit increments. The current L/S variants + are: `1/1`, `2/1`, `4/1`, `8/8`, `16/8`, `32/8`. +- arith L/8 - deterministic arithmetics. The fuzzer tries to subtract or add + small integers to 8-, 16-, and 32-bit values. The stepover is always 8 bits. +- interest L/8 - deterministic value overwrite. The fuzzer has a list of known + "interesting" 8-, 16-, and 32-bit values to try. The stepover is 8 bits. +- extras - deterministic injection of dictionary terms. This can be shown as + "user" or "auto", depending on whether the fuzzer is using a user-supplied + dictionary (`-x`) or an auto-created one. You will also see "over" or + "insert", depending on whether the dictionary words overwrite existing data or + are inserted by offsetting the remaining data to accommodate their length. +- havoc - a sort-of-fixed-length cycle with stacked random tweaks. The + operations attempted during this stage include bit flips, overwrites with + random and "interesting" integers, block deletion, block duplication, plus + assorted dictionary-related operations (if a dictionary is supplied in the + first place). +- splice - a last-resort strategy that kicks in after the first full queue cycle + with no new paths. It is equivalent to 'havoc', except that it first splices + together two random inputs from the queue at some arbitrarily selected + midpoint. +- sync - a stage used only when `-M` or `-S` is set (see + [parallel_fuzzing.md](parallel_fuzzing.md)). No real fuzzing is involved, but + the tool scans the output from other fuzzers and imports test cases as + necessary. The first time this is done, it may take several minutes or so. + +The remaining fields should be fairly self-evident: there's the exec count +progress indicator for the current stage, a global exec counter, and a benchmark +for the current program execution speed. This may fluctuate from one test case +to another, but the benchmark should be ideally over 500 execs/sec most of the +time - and if it stays below 100, the job will probably take very long. + +The fuzzer will explicitly warn you about slow targets, too. If this happens, +see the [perf_tips.md](perf_tips.md) file included with the fuzzer for ideas on +how to speed things up. + +### Findings in depth + +``` + +--------------------------------------+ + | favored paths : 879 (41.96%) | + | new edges on : 423 (20.19%) | + | total crashes : 0 (0 unique) | + | total tmouts : 24 (19 unique) | + +--------------------------------------+ +``` + +This gives you several metrics that are of interest mostly to complete nerds. +The section includes the number of paths that the fuzzer likes the most based on +a minimization algorithm baked into the code (these will get considerably more +air time), and the number of test cases that actually resulted in better edge +coverage (versus just pushing the branch hit counters up). There are also +additional, more detailed counters for crashes and timeouts. + +Note that the timeout counter is somewhat different from the hang counter; this +one includes all test cases that exceeded the timeout, even if they did not +exceed it by a margin sufficient to be classified as hangs. + +### Fuzzing strategy yields + +``` + +-----------------------------------------------------+ + | bit flips : 57/289k, 18/289k, 18/288k | + | byte flips : 0/36.2k, 4/35.7k, 7/34.6k | + | arithmetics : 53/2.54M, 0/537k, 0/55.2k | + | known ints : 8/322k, 12/1.32M, 10/1.70M | + | dictionary : 9/52k, 1/53k, 1/24k | + |havoc/splice : 1903/20.0M, 0/0 | + |py/custom/rq : unused, 53/2.54M, unused | + | trim/eff : 20.31%/9201, 17.05% | + +-----------------------------------------------------+ +``` + +This is just another nerd-targeted section keeping track of how many paths we +have netted, in proportion to the number of execs attempted, for each of the +fuzzing strategies discussed earlier on. This serves to convincingly validate +assumptions about the usefulness of the various approaches taken by afl-fuzz. + +The trim strategy stats in this section are a bit different than the rest. The +first number in this line shows the ratio of bytes removed from the input files; +the second one corresponds to the number of execs needed to achieve this goal. +Finally, the third number shows the proportion of bytes that, although not +possible to remove, were deemed to have no effect and were excluded from some of +the more expensive deterministic fuzzing steps. + +Note that when deterministic mutation mode is off (which is the default because +it is not very efficient) the first five lines display "disabled (default, +enable with -D)". + +Only what is activated will have counter shown. + +### Path geometry + +``` + +---------------------+ + | levels : 5 | + | pending : 1570 | + | pend fav : 583 | + | own finds : 0 | + | imported : 0 | + | stability : 100.00% | + +---------------------+ +``` + +The first field in this section tracks the path depth reached through the guided +fuzzing process. In essence: the initial test cases supplied by the user are +considered "level 1". The test cases that can be derived from that through +traditional fuzzing are considered "level 2"; the ones derived by using these as +inputs to subsequent fuzzing rounds are "level 3"; and so forth. The maximum +depth is therefore a rough proxy for how much value you're getting out of the +instrumentation-guided approach taken by afl-fuzz. + +The next field shows you the number of inputs that have not gone through any +fuzzing yet. The same stat is also given for "favored" entries that the fuzzer +really wants to get to in this queue cycle (the non-favored entries may have to +wait a couple of cycles to get their chance). + +Next, we have the number of new paths found during this fuzzing section and +imported from other fuzzer instances when doing parallelized fuzzing; and the +extent to which identical inputs appear to sometimes produce variable behavior +in the tested binary. + +That last bit is actually fairly interesting: it measures the consistency of +observed traces. If a program always behaves the same for the same input data, +it will earn a score of 100%. When the value is lower but still shown in purple, +the fuzzing process is unlikely to be negatively affected. If it goes into red, +you may be in trouble, since AFL will have difficulty discerning between +meaningful and "phantom" effects of tweaking the input file. + +Now, most targets will just get a 100% score, but when you see lower figures, +there are several things to look at: + +- The use of uninitialized memory in conjunction with some intrinsic sources of + entropy in the tested binary. Harmless to AFL, but could be indicative of a + security bug. +- Attempts to manipulate persistent resources, such as left over temporary files + or shared memory objects. This is usually harmless, but you may want to + double-check to make sure the program isn't bailing out prematurely. Running + out of disk space, SHM handles, or other global resources can trigger this, + too. +- Hitting some functionality that is actually designed to behave randomly. + Generally harmless. For example, when fuzzing sqlite, an input like `select + random();` will trigger a variable execution path. +- Multiple threads executing at once in semi-random order. This is harmless when + the 'stability' metric stays over 90% or so, but can become an issue if not. + Here's what to try: + * Use afl-clang-fast from [instrumentation](../instrumentation/) - it uses a + thread-local tracking model that is less prone to concurrency issues, + * See if the target can be compiled or run without threads. Common + `./configure` options include `--without-threads`, `--disable-pthreads`, or + `--disable-openmp`. + * Replace pthreads with GNU Pth (https://www.gnu.org/software/pth/), which + allows you to use a deterministic scheduler. +- In persistent mode, minor drops in the "stability" metric can be normal, + because not all the code behaves identically when re-entered; but major dips + may signify that the code within `__AFL_LOOP()` is not behaving correctly on + subsequent iterations (e.g., due to incomplete clean-up or reinitialization of + the state) and that most of the fuzzing effort goes to waste. + +The paths where variable behavior is detected are marked with a matching entry +in the `/queue/.state/variable_behavior/` directory, so you can look +them up easily. + +### CPU load + +``` + [cpu: 25%] +``` + +This tiny widget shows the apparent CPU utilization on the local system. It is +calculated by taking the number of processes in the "runnable" state, and then +comparing it to the number of logical cores on the system. + +If the value is shown in green, you are using fewer CPU cores than available on +your system and can probably parallelize to improve performance; for tips on how +to do that, see [parallel_fuzzing.md](parallel_fuzzing.md). + +If the value is shown in red, your CPU is *possibly* oversubscribed, and running +additional fuzzers may not give you any benefits. + +Of course, this benchmark is very simplistic; it tells you how many processes +are ready to run, but not how resource-hungry they may be. It also doesn't +distinguish between physical cores, logical cores, and virtualized CPUs; the +performance characteristics of each of these will differ quite a bit. + +If you want a more accurate measurement, you can run the `afl-gotcpu` utility +from the command line. + +## Interpreting output + +See [#understanding-the-status-screen](#understanding-the-status-screen) for +information on how to interpret the displayed stats and monitor the health of +the process. Be sure to consult this file especially if any UI elements are +highlighted in red. + +The fuzzing process will continue until you press Ctrl-C. At a minimum, you want +to allow the fuzzer to complete one queue cycle, which may take anywhere from a +couple of hours to a week or so. + +There are three subdirectories created within the output directory and updated +in real-time: + +- queue/ - test cases for every distinctive execution path, plus all the + starting files given by the user. This is the synthesized corpus + mentioned in section 2. + + Before using this corpus for any other purposes, you can shrink + it to a smaller size using the afl-cmin tool. The tool will find + a smaller subset of files offering equivalent edge coverage. + +- crashes/ - unique test cases that cause the tested program to receive a fatal + signal (e.g., SIGSEGV, SIGILL, SIGABRT). The entries are grouped by + the received signal. + +- hangs/ - unique test cases that cause the tested program to time out. The + default time limit before something is classified as a hang is the + larger of 1 second and the value of the -t parameter. The value can + be fine-tuned by setting AFL_HANG_TMOUT, but this is rarely + necessary. + +Crashes and hangs are considered "unique" if the associated execution paths +involve any state transitions not seen in previously-recorded faults. If a +single bug can be reached in multiple ways, there will be some count inflation +early in the process, but this should quickly taper off. + +The file names for crashes and hangs are correlated with the parent, non-faulting +queue entries. This should help with debugging. + +## Visualizing + +If you have gnuplot installed, you can also generate some pretty graphs for any +active fuzzing task using afl-plot. For an example of how this looks like, see +[https://lcamtuf.coredump.cx/afl/plot/](https://lcamtuf.coredump.cx/afl/plot/). + +You can also manually build and install afl-plot-ui, which is a helper utility +for showing the graphs generated by afl-plot in a graphical window using GTK. +You can build and install it as follows: + +```shell +sudo apt install libgtk-3-0 libgtk-3-dev pkg-config +cd utils/plot_ui +make +cd ../../ +sudo make install +``` + + +### Addendum: status and plot files + +For unattended operation, some of the key status screen information can be also +found in a machine-readable format in the fuzzer_stats file in the output +directory. This includes: + +- `start_time` - unix time indicating the start time of afl-fuzz +- `last_update` - unix time corresponding to the last update of this file +- `run_time` - run time in seconds to the last update of this file +- `fuzzer_pid` - PID of the fuzzer process +- `cycles_done` - queue cycles completed so far +- `cycles_wo_finds` - number of cycles without any new paths found +- `execs_done` - number of execve() calls attempted +- `execs_per_sec` - overall number of execs per second +- `paths_total` - total number of entries in the queue +- `paths_favored` - number of queue entries that are favored +- `paths_found` - number of entries discovered through local fuzzing +- `paths_imported` - number of entries imported from other instances +- `max_depth` - number of levels in the generated data set +- `cur_path` - currently processed entry number +- `pending_favs` - number of favored entries still waiting to be fuzzed +- `pending_total` - number of all entries waiting to be fuzzed +- `variable_paths` - number of test cases showing variable behavior +- `stability` - percentage of bitmap bytes that behave consistently +- `bitmap_cvg` - percentage of edge coverage found in the map so far +- `unique_crashes` - number of unique crashes recorded +- `unique_hangs` - number of unique hangs encountered +- `last_path` - seconds since the last path was found +- `last_crash` - seconds since the last crash was found +- `last_hang` - seconds since the last hang was found +- `execs_since_crash` - execs since the last crash was found +- `exec_timeout` - the -t command line value +- `slowest_exec_ms` - real time of the slowest execution in ms +- `peak_rss_mb` - max rss usage reached during fuzzing in MB +- `edges_found` - how many edges have been found +- `var_byte_count` - how many edges are non-deterministic +- `afl_banner` - banner text (e.g. the target name) +- `afl_version` - the version of AFL used +- `target_mode` - default, persistent, qemu, unicorn, non-instrumented +- `command_line` - full command line used for the fuzzing session + +Most of these map directly to the UI elements discussed earlier on. + +On top of that, you can also find an entry called `plot_data`, containing a +plottable history for most of these fields. If you have gnuplot installed, you +can turn this into a nice progress report with the included `afl-plot` tool. + +### Addendum: automatically sending metrics with StatsD + +In a CI environment or when running multiple fuzzers, it can be tedious to log +into each of them or deploy scripts to read the fuzzer statistics. Using +`AFL_STATSD` (and the other related environment variables `AFL_STATSD_HOST`, +`AFL_STATSD_PORT`, `AFL_STATSD_TAGS_FLAVOR`) you can automatically send metrics +to your favorite StatsD server. Depending on your StatsD server, you will be +able to monitor, trigger alerts, or perform actions based on these metrics (e.g: +alert on slow exec/s for a new build, threshold of crashes, time since last +crash > X, etc). + +The selected metrics are a subset of all the metrics found in the status and in +the plot file. The list is the following: `cycle_done`, `cycles_wo_finds`, +`execs_done`,`execs_per_sec`, `paths_total`, `paths_favored`, `paths_found`, +`paths_imported`, `max_depth`, `cur_path`, `pending_favs`, `pending_total`, +`variable_paths`, `unique_crashes`, `unique_hangs`, `total_crashes`, +`slowest_exec_ms`, `edges_found`, `var_byte_count`, `havoc_expansion`. Their +definitions can be found in the addendum above. + +When using multiple fuzzer instances with StatsD, it is *strongly* recommended +to setup the flavor (AFL_STATSD_TAGS_FLAVOR) to match your StatsD server. This +will allow you to see individual fuzzer performance, detect bad ones, see the +progress of each strategy... \ No newline at end of file diff --git a/docs/custom_mutators.md b/docs/custom_mutators.md index 8b5a4068..b1dfd309 100644 --- a/docs/custom_mutators.md +++ b/docs/custom_mutators.md @@ -127,9 +127,9 @@ def deinit(): # optional for Python - `describe` (optional): - When this function is called, it shall describe the current testcase, + When this function is called, it shall describe the current test case, generated by the last mutation. This will be called, for example, - to name the written testcase file after a crash occurred. + to name the written test case file after a crash occurred. Using it can help to reproduce crashing mutations. - `havoc_mutation` and `havoc_mutation_probability` (optional): @@ -224,7 +224,7 @@ Optionally, the following environment variables are supported: - `AFL_CUSTOM_MUTATOR_ONLY` - Disable all other mutation stages. This can prevent broken testcases + Disable all other mutation stages. This can prevent broken test cases (those that your Python module can't work with anymore) to fill up your queue. Best combined with a custom trimming routine (see below) because trimming can cause the same test breakage like havoc and splice. diff --git a/docs/env_variables.md b/docs/env_variables.md index 65cca0dc..34318cd4 100644 --- a/docs/env_variables.md +++ b/docs/env_variables.md @@ -306,8 +306,9 @@ checks or alter some of the more exotic semantics of the tool: exit soon after the first crash is found. - `AFL_CMPLOG_ONLY_NEW` will only perform the expensive cmplog feature for - newly found testcases and not for testcases that are loaded on startup (`-i - in`). This is an important feature to set when resuming a fuzzing session. + newly found test cases and not for test cases that are loaded on startup + (`-i in`). This is an important feature to set when resuming a fuzzing + session. - Setting `AFL_CRASH_EXITCODE` sets the exit code AFL treats as crash. For example, if `AFL_CRASH_EXITCODE='-1'` is set, each input resulting in a `-1` @@ -447,8 +448,8 @@ checks or alter some of the more exotic semantics of the tool: - If you are using persistent mode (you should, see [instrumentation/README.persistent_mode.md](../instrumentation/README.persistent_mode.md)), - some targets keep inherent state due which a detected crash testcase does - not crash the target again when the testcase is given. To be able to still + some targets keep inherent state due which a detected crash test case does + not crash the target again when the test case is given. To be able to still re-trigger these crashes, you can use the `AFL_PERSISTENT_RECORD` variable with a value of how many previous fuzz cases to keep prio a crash. If set to e.g. 10, then the 9 previous inputs are written to out/default/crashes as diff --git a/docs/features.md b/docs/features.md index f44e32ff..05670e6f 100644 --- a/docs/features.md +++ b/docs/features.md @@ -17,7 +17,7 @@ | Context Coverage | | x(6) | | | | | | | Auto Dictionary | | x(7) | | | | | | | Snapshot LKM Support | | (x)(8) | (x)(8) | | (x)(5) | | | - | Shared Memory Testcases | | x | x | x86[_64]/arm64 | x | x | | + | Shared Memory Test cases | | x | x | x86[_64]/arm64 | x | x | | 1. default for LLVM >= 9.0, env var for older version due an efficiency bug in previous llvm versions 2. GCC creates non-performant code, hence it is disabled in gcc_plugin diff --git a/docs/fuzzing_in_depth.md b/docs/fuzzing_in_depth.md index 5306cbef..5b4a9df7 100644 --- a/docs/fuzzing_in_depth.md +++ b/docs/fuzzing_in_depth.md @@ -1,24 +1,25 @@ # Fuzzing with AFL++ The following describes how to fuzz with a target if source code is available. -If you have a binary-only target please skip to [#Instrumenting binary-only apps](#Instrumenting binary-only apps) +If you have a binary-only target, please go to +[fuzzing_binary-only_targets.md](fuzzing_binary-only_targets.md). -Fuzzing source code is a three-step process. +Fuzzing source code is a three-step process: 1. Compile the target with a special compiler that prepares the target to be fuzzed efficiently. This step is called "instrumenting a target". 2. Prepare the fuzzing by selecting and optimizing the input corpus for the target. -3. Perform the fuzzing of the target by randomly mutating input and assessing - if a generated input was processed in a new path in the target binary. +3. Perform the fuzzing of the target by randomly mutating input and assessing if + a generated input was processed in a new path in the target binary. ### 1. Instrumenting that target #### a) Selecting the best AFL++ compiler for instrumenting the target AFL++ comes with a central compiler `afl-cc` that incorporates various different -kinds of compiler targets and and instrumentation options. -The following evaluation flow will help you to select the best possible. +kinds of compiler targets and and instrumentation options. The following +evaluation flow will help you to select the best possible. It is highly recommended to have the newest llvm version possible installed, anything below 9 is not recommended. @@ -51,132 +52,131 @@ anything below 9 is not recommended. Clickable README links for the chosen compiler: - * [LTO mode - afl-clang-lto](../instrumentation/README.lto.md) - * [LLVM mode - afl-clang-fast](../instrumentation/README.llvm.md) - * [GCC_PLUGIN mode - afl-gcc-fast](../instrumentation/README.gcc_plugin.md) - * GCC/CLANG modes (afl-gcc/afl-clang) have no README as they have no own features +* [LTO mode - afl-clang-lto](../instrumentation/README.lto.md) +* [LLVM mode - afl-clang-fast](../instrumentation/README.llvm.md) +* [GCC_PLUGIN mode - afl-gcc-fast](../instrumentation/README.gcc_plugin.md) +* GCC/CLANG modes (afl-gcc/afl-clang) have no README as they have no own + features You can select the mode for the afl-cc compiler by: - 1. use a symlink to afl-cc: afl-gcc, afl-g++, afl-clang, afl-clang++, - afl-clang-fast, afl-clang-fast++, afl-clang-lto, afl-clang-lto++, - afl-gcc-fast, afl-g++-fast (recommended!) - 2. using the environment variable AFL_CC_COMPILER with MODE - 3. passing --afl-MODE command line options to the compiler via CFLAGS/CXXFLAGS/CPPFLAGS +1. use a symlink to afl-cc: afl-gcc, afl-g++, afl-clang, afl-clang++, + afl-clang-fast, afl-clang-fast++, afl-clang-lto, afl-clang-lto++, + afl-gcc-fast, afl-g++-fast (recommended!) +2. using the environment variable AFL_CC_COMPILER with MODE +3. passing --afl-MODE command line options to the compiler via + CFLAGS/CXXFLAGS/CPPFLAGS MODE can be one of: LTO (afl-clang-lto*), LLVM (afl-clang-fast*), GCC_PLUGIN (afl-g*-fast) or GCC (afl-gcc/afl-g++) or CLANG(afl-clang/afl-clang++). -Because no AFL specific command-line options are accepted (beside the ---afl-MODE command), the compile-time tools make fairly broad use of environment -variables, which can be listed with `afl-cc -hh` or by reading [env_variables.md](env_variables.md). +Because no AFL specific command-line options are accepted (beside the --afl-MODE +command), the compile-time tools make fairly broad use of environment variables, +which can be listed with `afl-cc -hh` or by reading +[env_variables.md](env_variables.md). #### b) Selecting instrumentation options -The following options are available when you instrument with LTO mode (afl-clang-fast/afl-clang-lto): - - * Splitting integer, string, float and switch comparisons so AFL++ can easier - solve these. This is an important option if you do not have a very good - and large input corpus. This technique is called laf-intel or COMPCOV. - To use this set the following environment variable before compiling the - target: `export AFL_LLVM_LAF_ALL=1` - You can read more about this in [instrumentation/README.laf-intel.md](../instrumentation/README.laf-intel.md) - * A different technique (and usually a better one than laf-intel) is to - instrument the target so that any compare values in the target are sent to - AFL++ which then tries to put these values into the fuzzing data at different - locations. This technique is very fast and good - if the target does not - transform input data before comparison. Therefore this technique is called - `input to state` or `redqueen`. - If you want to use this technique, then you have to compile the target - twice, once specifically with/for this mode by setting `AFL_LLVM_CMPLOG=1`, - and pass this binary to afl-fuzz via the `-c` parameter. - Note that you can compile also just a cmplog binary and use that for both - however there will be a performance penality. - You can read more about this in [instrumentation/README.cmplog.md](../instrumentation/README.cmplog.md) - -If you use LTO, LLVM or GCC_PLUGIN mode (afl-clang-fast/afl-clang-lto/afl-gcc-fast) -you have the option to selectively only instrument parts of the target that you -are interested in: - - * To instrument only those parts of the target that you are interested in - create a file with all the filenames of the source code that should be - instrumented. - For afl-clang-lto and afl-gcc-fast - or afl-clang-fast if a mode other than - DEFAULT/PCGUARD is used or you have llvm > 10.0.0 - just put one - filename or function per line (no directory information necessary for - filenames9, and either set `export AFL_LLVM_ALLOWLIST=allowlist.txt` **or** - `export AFL_LLVM_DENYLIST=denylist.txt` - depending on if you want per - default to instrument unless noted (DENYLIST) or not perform instrumentation - unless requested (ALLOWLIST). - **NOTE:** During optimization functions might be inlined and then would not match! - See [instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md) +The following options are available when you instrument with LTO mode +(afl-clang-fast/afl-clang-lto): + +* Splitting integer, string, float and switch comparisons so AFL++ can easier + solve these. This is an important option if you do not have a very good and + large input corpus. This technique is called laf-intel or COMPCOV. To use this + set the following environment variable before compiling the target: `export + AFL_LLVM_LAF_ALL=1` You can read more about this in + [instrumentation/README.laf-intel.md](../instrumentation/README.laf-intel.md). +* A different technique (and usually a better one than laf-intel) is to + instrument the target so that any compare values in the target are sent to + AFL++ which then tries to put these values into the fuzzing data at different + locations. This technique is very fast and good - if the target does not + transform input data before comparison. Therefore this technique is called + `input to state` or `redqueen`. If you want to use this technique, then you + have to compile the target twice, once specifically with/for this mode by + setting `AFL_LLVM_CMPLOG=1`, and pass this binary to afl-fuzz via the `-c` + parameter. Note that you can compile also just a cmplog binary and use that + for both however there will be a performance penality. You can read more about + this in + [instrumentation/README.cmplog.md](../instrumentation/README.cmplog.md). + +If you use LTO, LLVM or GCC_PLUGIN mode +(afl-clang-fast/afl-clang-lto/afl-gcc-fast) you have the option to selectively +only instrument parts of the target that you are interested in: + +* To instrument only those parts of the target that you are interested in create + a file with all the filenames of the source code that should be instrumented. + For afl-clang-lto and afl-gcc-fast - or afl-clang-fast if a mode other than + DEFAULT/PCGUARD is used or you have llvm > 10.0.0 - just put one filename or + function per line (no directory information necessary for filenames9, and + either set `export AFL_LLVM_ALLOWLIST=allowlist.txt` **or** `export + AFL_LLVM_DENYLIST=denylist.txt` - depending on if you want per default to + instrument unless noted (DENYLIST) or not perform instrumentation unless + requested (ALLOWLIST). **NOTE:** During optimization functions might be + inlined and then would not match! See + [instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md) There are many more options and modes available however these are most of the time less effective. See: - * [instrumentation/README.ctx.md](../instrumentation/README.ctx.md) - * [instrumentation/README.ngram.md](../instrumentation/README.ngram.md) +* [instrumentation/README.ctx.md](../instrumentation/README.ctx.md) +* [instrumentation/README.ngram.md](../instrumentation/README.ngram.md) AFL++ performs "never zero" counting in its bitmap. You can read more about this here: - * [instrumentation/README.neverzero.md](../instrumentation/README.neverzero.md) +* [instrumentation/README.neverzero.md](../instrumentation/README.neverzero.md) #### c) Sanitizers -It is possible to use sanitizers when instrumenting targets for fuzzing, -which allows you to find bugs that would not necessarily result in a crash. +It is possible to use sanitizers when instrumenting targets for fuzzing, which +allows you to find bugs that would not necessarily result in a crash. Note that sanitizers have a huge impact on CPU (= less executions per second) -and RAM usage. Also you should only run one afl-fuzz instance per sanitizer type. -This is enough because a use-after-free bug will be picked up, e.g. by -ASAN (address sanitizer) anyway when syncing to other fuzzing instances, -so not all fuzzing instances need to be instrumented with ASAN. +and RAM usage. Also you should only run one afl-fuzz instance per sanitizer +type. This is enough because a use-after-free bug will be picked up, e.g. by +ASAN (address sanitizer) anyway when syncing to other fuzzing instances, so not +all fuzzing instances need to be instrumented with ASAN. The following sanitizers have built-in support in AFL++: - * ASAN = Address SANitizer, finds memory corruption vulnerabilities like - use-after-free, NULL pointer dereference, buffer overruns, etc. - Enabled with `export AFL_USE_ASAN=1` before compiling. - * MSAN = Memory SANitizer, finds read access to uninitialized memory, eg. - a local variable that is defined and read before it is even set. - Enabled with `export AFL_USE_MSAN=1` before compiling. - * UBSAN = Undefined Behaviour SANitizer, finds instances where - by the - C and C++ standards - undefined behaviour happens, e.g. adding two - signed integers together where the result is larger than a signed integer - can hold. - Enabled with `export AFL_USE_UBSAN=1` before compiling. - * CFISAN = Control Flow Integrity SANitizer, finds instances where the - control flow is found to be illegal. Originally this was rather to - prevent return oriented programming exploit chains from functioning, - in fuzzing this is mostly reduced to detecting type confusion - vulnerabilities - which is however one of the most important and dangerous - C++ memory corruption classes! - Enabled with `export AFL_USE_CFISAN=1` before compiling. - * TSAN = Thread SANitizer, finds thread race conditions. - Enabled with `export AFL_USE_TSAN=1` before compiling. - * LSAN = Leak SANitizer, finds memory leaks in a program. This is not really - a security issue, but for developers this can be very valuable. - Note that unlike the other sanitizers above this needs - `__AFL_LEAK_CHECK();` added to all areas of the target source code where you - find a leak check necessary! - Enabled with `export AFL_USE_LSAN=1` before compiling. - -It is possible to further modify the behaviour of the sanitizers at run-time -by setting `ASAN_OPTIONS=...`, `LSAN_OPTIONS` etc. - the available parameters -can be looked up in the sanitizer documentation of llvm/clang. -afl-fuzz however requires some specific parameters important for fuzzing to be -set. If you want to set your own, it might bail and report what it is missing. +* ASAN = Address SANitizer, finds memory corruption vulnerabilities like + use-after-free, NULL pointer dereference, buffer overruns, etc. Enabled with + `export AFL_USE_ASAN=1` before compiling. +* MSAN = Memory SANitizer, finds read access to uninitialized memory, eg. a + local variable that is defined and read before it is even set. Enabled with + `export AFL_USE_MSAN=1` before compiling. +* UBSAN = Undefined Behaviour SANitizer, finds instances where - by the C and + C++ standards - undefined behaviour happens, e.g. adding two signed integers + together where the result is larger than a signed integer can hold. Enabled + with `export AFL_USE_UBSAN=1` before compiling. +* CFISAN = Control Flow Integrity SANitizer, finds instances where the control + flow is found to be illegal. Originally this was rather to prevent return + oriented programming exploit chains from functioning, in fuzzing this is + mostly reduced to detecting type confusion vulnerabilities - which is, + however, one of the most important and dangerous C++ memory corruption + classes! Enabled with `export AFL_USE_CFISAN=1` before compiling. +* TSAN = Thread SANitizer, finds thread race conditions. Enabled with `export + AFL_USE_TSAN=1` before compiling. +* LSAN = Leak SANitizer, finds memory leaks in a program. This is not really a + security issue, but for developers this can be very valuable. Note that unlike + the other sanitizers above this needs `__AFL_LEAK_CHECK();` added to all areas + of the target source code where you find a leak check necessary! Enabled with + `export AFL_USE_LSAN=1` before compiling. + +It is possible to further modify the behaviour of the sanitizers at run-time by +setting `ASAN_OPTIONS=...`, `LSAN_OPTIONS` etc. - the available parameters can +be looked up in the sanitizer documentation of llvm/clang. afl-fuzz, however, +requires some specific parameters important for fuzzing to be set. If you want +to set your own, it might bail and report what it is missing. Note that some sanitizers cannot be used together, e.g. ASAN and MSAN, and others often cannot work together because of target weirdness, e.g. ASAN and CFISAN. You might need to experiment which sanitizers you can combine in a -target (which means more instances can be run without a sanitized target, -which is more effective). +target (which means more instances can be run without a sanitized target, which +is more effective). #### d) Modify the target -If the target has features that make fuzzing more difficult, e.g. -checksums, HMAC, etc. then modify the source code so that checks for these -values are removed. -This can even be done safely for source code used in operational products -by eliminating these checks within these AFL specific blocks: +If the target has features that make fuzzing more difficult, e.g. checksums, +HMAC, etc. then modify the source code so that checks for these values are +removed. This can even be done safely for source code used in operational +products by eliminating these checks within these AFL specific blocks: ``` #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION @@ -193,25 +193,24 @@ All AFL++ compilers will set this preprocessor definition automatically. In this step the target source code is compiled so that it can be fuzzed. Basically you have to tell the target build system that the selected AFL++ -compiler is used. Also - if possible - you should always configure the -build system such that the target is compiled statically and not dynamically. -How to do this is described below. +compiler is used. Also - if possible - you should always configure the build +system such that the target is compiled statically and not dynamically. How to +do this is described below. -The #1 rule when instrumenting a target is: avoid instrumenting shared -libraries at all cost. You would need to set LD_LIBRARY_PATH to point to -these, you could accidently type "make install" and install them system wide - -so don't. Really don't. -**Always compile libraries you want to have instrumented as static and link -these to the target program!** +The #1 rule when instrumenting a target is: avoid instrumenting shared libraries +at all cost. You would need to set LD_LIBRARY_PATH to point to these, you could +accidentally type "make install" and install them system wide - so don't. Really +don't. **Always compile libraries you want to have instrumented as static and +link these to the target program!** Then build the target. (Usually with `make`) **NOTES** -1. sometimes configure and build systems are fickle and do not like - stderr output (and think this means a test failure) - which is something - AFL++ likes to do to show statistics. It is recommended to disable AFL++ - instrumentation reporting via `export AFL_QUIET=1`. +1. sometimes configure and build systems are fickle and do not like stderr + output (and think this means a test failure) - which is something AFL++ likes + to do to show statistics. It is recommended to disable AFL++ instrumentation + reporting via `export AFL_QUIET=1`. 2. sometimes configure and build systems error on warnings - these should be disabled (e.g. `--disable-werror` for some configure scripts). @@ -249,41 +248,46 @@ Sometimes cmake and configure do not pick up the AFL++ compiler, or the ranlib/ar that is needed - because this was just not foreseen by the developer of the target. Or they have non-standard options. Figure out if there is a non-standard way to set this, otherwise set up the build normally and edit the -generated build environment afterwards manually to point it to the right compiler -(and/or ranlib and ar). +generated build environment afterwards manually to point it to the right +compiler (and/or ranlib and ar). #### f) Better instrumentation If you just fuzz a target program as-is you are wasting a great opportunity for much more fuzzing speed. -This variant requires the usage of afl-clang-lto, afl-clang-fast or afl-gcc-fast. +This variant requires the usage of afl-clang-lto, afl-clang-fast or +afl-gcc-fast. -It is the so-called `persistent mode`, which is much, much faster but -requires that you code a source file that is specifically calling the target -functions that you want to fuzz, plus a few specific AFL++ functions around -it. See [instrumentation/README.persistent_mode.md](../instrumentation/README.persistent_mode.md) for details. +It is the so-called `persistent mode`, which is much, much faster but requires +that you code a source file that is specifically calling the target functions +that you want to fuzz, plus a few specific AFL++ functions around it. See +[instrumentation/README.persistent_mode.md](../instrumentation/README.persistent_mode.md) +for details. -Basically if you do not fuzz a target in persistent mode then you are just -doing it for a hobby and not professionally :-). +Basically if you do not fuzz a target in persistent mode then you are just doing +it for a hobby and not professionally :-). #### g) libfuzzer fuzzer harnesses with LLVMFuzzerTestOneInput() libfuzzer `LLVMFuzzerTestOneInput()` harnesses are the defacto standard for fuzzing, and they can be used with AFL++ (and honggfuzz) as well! Compiling them is as simple as: + ``` afl-clang-fast++ -fsanitize=fuzzer -o harness harness.cpp targetlib.a ``` + You can even use advanced libfuzzer features like `FuzzedDataProvider`, `LLVMFuzzerMutate()` etc. and they will work! The generated binary is fuzzed with afl-fuzz like any other fuzz target. Bonus: the target is already optimized for fuzzing due to persistent mode and -shared-memory testcases and hence gives you the fastest speed possible. +shared-memory test cases and hence gives you the fastest speed possible. -For more information see [utils/aflpp_driver/README.md](../utils/aflpp_driver/README.md) +For more information, see +[utils/aflpp_driver/README.md](../utils/aflpp_driver/README.md). ### 2. Preparing the fuzzing campaign @@ -294,8 +298,8 @@ target as possible improves the efficiency a lot. Try to gather valid inputs for the target from wherever you can. E.g. if it is the PNG picture format try to find as many png files as possible, e.g. from -reported bugs, test suites, random downloads from the internet, unit test -case data - from all kind of PNG software. +reported bugs, test suites, random downloads from the internet, unit test case +data - from all kind of PNG software. If the input format is not known, you can also modify a target program to write normal data it receives and processes to a file and use these. @@ -319,10 +323,9 @@ This step is highly recommended! #### c) Minimizing all corpus files -The shorter the input files that still traverse the same path -within the target, the better the fuzzing will be. This minimization -is done with `afl-tmin` however it is a long process as this has to -be done for every file: +The shorter the input files that still traverse the same path within the target, +the better the fuzzing will be. This minimization is done with `afl-tmin` +however it is a long process as this has to be done for every file: ``` mkdir input @@ -332,8 +335,8 @@ for i in *; do done ``` -This step can also be parallelized, e.g. with `parallel`. -Note that this step is rather optional though. +This step can also be parallelized, e.g. with `parallel`. Note that this step is +rather optional though. #### Done! @@ -343,10 +346,9 @@ to be used in fuzzing! :-) ### 3. Fuzzing the target -In this final step we fuzz the target. -There are not that many important options to run the target - unless you want -to use many CPU cores/threads for the fuzzing, which will make the fuzzing much -more useful. +In this final step we fuzz the target. There are not that many important options +to run the target - unless you want to use many CPU cores/threads for the +fuzzing, which will make the fuzzing much more useful. If you just use one CPU for fuzzing, then you are fuzzing just for fun and not seriously :-) @@ -355,19 +357,19 @@ seriously :-) Before you do even a test run of afl-fuzz execute `sudo afl-system-config` (on the host if you execute afl-fuzz in a docker container). This reconfigures the -system for optimal speed - which afl-fuzz checks and bails otherwise. -Set `export AFL_SKIP_CPUFREQ=1` for afl-fuzz to skip this check if you cannot -run afl-system-config with root privileges on the host for whatever reason. +system for optimal speed - which afl-fuzz checks and bails otherwise. Set +`export AFL_SKIP_CPUFREQ=1` for afl-fuzz to skip this check if you cannot run +afl-system-config with root privileges on the host for whatever reason. Note there is also `sudo afl-persistent-config` which sets additional permanent boot options for a much better fuzzing performance. Note that both scripts improve your fuzzing performance but also decrease your -system protection against attacks! So set strong firewall rules and only -expose SSH as a network service if you use these (which is highly recommended). +system protection against attacks! So set strong firewall rules and only expose +SSH as a network service if you use these (which is highly recommended). -If you have an input corpus from step 2 then specify this directory with the `-i` -option. Otherwise create a new directory and create a file with any content +If you have an input corpus from step 2 then specify this directory with the +`-i` option. Otherwise create a new directory and create a file with any content as test data in there. If you do not want anything special, the defaults are already usually best, @@ -387,36 +389,37 @@ same as the afl-fuzz -M/-S naming :-) For more information on screen or tmux please check their documentation. If you need to stop and re-start the fuzzing, use the same command line options -(or even change them by selecting a different power schedule or another -mutation mode!) and switch the input directory with a dash (`-`): +(or even change them by selecting a different power schedule or another mutation +mode!) and switch the input directory with a dash (`-`): `afl-fuzz -i - -o output -- bin/target -d @@` -Memory limits are not enforced by afl-fuzz by default and the system may run -out of memory. You can decrease the memory with the `-m` option, the value is -in MB. If this is too small for the target, you can usually see this by -afl-fuzz bailing with the message that it could not connect to the forkserver. +Memory limits are not enforced by afl-fuzz by default and the system may run out +of memory. You can decrease the memory with the `-m` option, the value is in MB. +If this is too small for the target, you can usually see this by afl-fuzz +bailing with the message that it could not connect to the forkserver. -Adding a dictionary is helpful. See the directory [dictionaries/](../dictionaries/) if -something is already included for your data format, and tell afl-fuzz to load -that dictionary by adding `-x dictionaries/FORMAT.dict`. With afl-clang-lto -you have an autodictionary generation for which you need to do nothing except -to use afl-clang-lto as the compiler. You also have the option to generate -a dictionary yourself, see [utils/libtokencap/README.md](../utils/libtokencap/README.md). +Adding a dictionary is helpful. See the directory +[dictionaries/](../dictionaries/) if something is already included for your data +format, and tell afl-fuzz to load that dictionary by adding `-x +dictionaries/FORMAT.dict`. With afl-clang-lto you have an autodictionary +generation for which you need to do nothing except to use afl-clang-lto as the +compiler. You also have the option to generate a dictionary yourself, see +[utils/libtokencap/README.md](../utils/libtokencap/README.md). afl-fuzz has a variety of options that help to workaround target quirks like -specific locations for the input file (`-f`), performing deterministic -fuzzing (`-D`) and many more. Check out `afl-fuzz -h`. +specific locations for the input file (`-f`), performing deterministic fuzzing +(`-D`) and many more. Check out `afl-fuzz -h`. We highly recommend that you set a memory limit for running the target with `-m` -which defines the maximum memory in MB. This prevents a potential -out-of-memory problem for your system plus helps you detect missing `malloc()` -failure handling in the target. -Play around with various -m values until you find one that safely works for all -your input seeds (if you have good ones and then double or quadrouple that. +which defines the maximum memory in MB. This prevents a potential out-of-memory +problem for your system plus helps you detect missing `malloc()` failure +handling in the target. Play around with various -m values until you find one +that safely works for all your input seeds (if you have good ones and then +double or quadruple that. -By default afl-fuzz never stops fuzzing. To terminate AFL++ simply press Control-C -or send a signal SIGINT. You can limit the number of executions or approximate runtime -in seconds with options also. +By default afl-fuzz never stops fuzzing. To terminate AFL++ simply press +Control-C or send a signal SIGINT. You can limit the number of executions or +approximate runtime in seconds with options also. When you start afl-fuzz you will see a user interface that shows what the status is: @@ -426,67 +429,67 @@ All labels are explained in [status_screen.md](status_screen.md). #### b) Using multiple cores -If you want to seriously fuzz then use as many cores/threads as possible to -fuzz your target. +If you want to seriously fuzz then use as many cores/threads as possible to fuzz +your target. On the same machine - due to the design of how AFL++ works - there is a maximum -number of CPU cores/threads that are useful, use more and the overall performance -degrades instead. This value depends on the target, and the limit is between 32 -and 64 cores per machine. +number of CPU cores/threads that are useful, use more and the overall +performance degrades instead. This value depends on the target, and the limit is +between 32 and 64 cores per machine. If you have the RAM, it is highly recommended run the instances with a caching -of the testcases. Depending on the average testcase size (and those found -during fuzzing) and their number, a value between 50-500MB is recommended. -You can set the cache size (in MB) by setting the environment variable `AFL_TESTCACHE_SIZE`. +of the test cases. Depending on the average test case size (and those found +during fuzzing) and their number, a value between 50-500MB is recommended. You +can set the cache size (in MB) by setting the environment variable +`AFL_TESTCACHE_SIZE`. -There should be one main fuzzer (`-M main-$HOSTNAME` option) and as many secondary -fuzzers (eg `-S variant1`) as you have cores that you use. -Every -M/-S entry needs a unique name (that can be whatever), however the same --o output directory location has to be used for all instances. +There should be one main fuzzer (`-M main-$HOSTNAME` option) and as many +secondary fuzzers (e.g. `-S variant1`) as you have cores that you use. Every +-M/-S entry needs a unique name (that can be whatever), however, the same -o +output directory location has to be used for all instances. For every secondary fuzzer there should be a variation, e.g.: - * one should fuzz the target that was compiled differently: with sanitizers - activated (`export AFL_USE_ASAN=1 ; export AFL_USE_UBSAN=1 ; - export AFL_USE_CFISAN=1`) - * one or two should fuzz the target with CMPLOG/redqueen (see above), at - least one cmplog instance should follow transformations (`-l AT`) - * one to three fuzzers should fuzz a target compiled with laf-intel/COMPCOV - (see above). Important note: If you run more than one laf-intel/COMPCOV - fuzzer and you want them to share their intermediate results, the main - fuzzer (`-M`) must be one of the them! (Although this is not really - recommended.) +* one should fuzz the target that was compiled differently: with sanitizers + activated (`export AFL_USE_ASAN=1 ; export AFL_USE_UBSAN=1 ; export + AFL_USE_CFISAN=1`) +* one or two should fuzz the target with CMPLOG/redqueen (see above), at least + one cmplog instance should follow transformations (`-l AT`) +* one to three fuzzers should fuzz a target compiled with laf-intel/COMPCOV (see + above). Important note: If you run more than one laf-intel/COMPCOV fuzzer and + you want them to share their intermediate results, the main fuzzer (`-M`) must + be one of the them! (Although this is not really recommended.) All other secondaries should be used like this: - * A quarter to a third with the MOpt mutator enabled: `-L 0` - * run with a different power schedule, recommended are: - `fast (default), explore, coe, lin, quad, exploit and rare` - which you can set with e.g. `-p explore` - * a few instances should use the old queue cycling with `-Z` +* a quarter to a third with the MOpt mutator enabled: `-L 0` +* run with a different power schedule, recommended are: + `fast (default), explore, coe, lin, quad, exploit and rare` which you can set + with e.g. `-p explore` +* a few instances should use the old queue cycling with `-Z` -Also it is recommended to set `export AFL_IMPORT_FIRST=1` to load testcases +Also, it is recommended to set `export AFL_IMPORT_FIRST=1` to load test cases from other fuzzers in the campaign first. If you have a large corpus, a corpus from a previous run or are fuzzing in a CI, then also set `export AFL_CMPLOG_ONLY_NEW=1` and `export AFL_FAST_CAL=1`. -You can also use different fuzzers. -If you are using AFL spinoffs or AFL conforming fuzzers, then just use the -same -o directory and give it a unique `-S` name. -Examples are: - * [Fuzzolic](https://github.com/season-lab/fuzzolic) - * [symcc](https://github.com/eurecom-s3/symcc/) - * [Eclipser](https://github.com/SoftSec-KAIST/Eclipser/) - * [AFLsmart](https://github.com/aflsmart/aflsmart) - * [FairFuzz](https://github.com/carolemieux/afl-rb) - * [Neuzz](https://github.com/Dongdongshe/neuzz) - * [Angora](https://github.com/AngoraFuzzer/Angora) - -A long list can be found at [https://github.com/Microsvuln/Awesome-AFL](https://github.com/Microsvuln/Awesome-AFL) - -However you can also sync AFL++ with honggfuzz, libfuzzer with `-entropic=1`, etc. -Just show the main fuzzer (-M) with the `-F` option where the queue/work -directory of a different fuzzer is, e.g. `-F /src/target/honggfuzz`. -Using honggfuzz (with `-n 1` or `-n 2`) and libfuzzer in parallel is highly +You can also use different fuzzers. If you are using AFL spinoffs or AFL +conforming fuzzers, then just use the same -o directory and give it a unique +`-S` name. Examples are: +* [Fuzzolic](https://github.com/season-lab/fuzzolic) +* [symcc](https://github.com/eurecom-s3/symcc/) +* [Eclipser](https://github.com/SoftSec-KAIST/Eclipser/) +* [AFLsmart](https://github.com/aflsmart/aflsmart) +* [FairFuzz](https://github.com/carolemieux/afl-rb) +* [Neuzz](https://github.com/Dongdongshe/neuzz) +* [Angora](https://github.com/AngoraFuzzer/Angora) + +A long list can be found at +[https://github.com/Microsvuln/Awesome-AFL](https://github.com/Microsvuln/Awesome-AFL). + +However, you can also sync AFL++ with honggfuzz, libfuzzer with `-entropic=1`, +etc. Just show the main fuzzer (-M) with the `-F` option where the queue/work +directory of a different fuzzer is, e.g. `-F /src/target/honggfuzz`. Using +honggfuzz (with `-n 1` or `-n 2`) and libfuzzer in parallel is highly recommended! #### c) Using multiple machines for fuzzing @@ -498,26 +501,24 @@ instance per server, and that its name is unique, hence the recommendation for `-M main-$HOSTNAME`. Now there are three strategies on how you can sync between the servers: - * never: sounds weird, but this makes every server an island and has the - chance the each follow different paths into the target. You can make - this even more interesting by even giving different seeds to each server. - * regularly (~4h): this ensures that all fuzzing campaigns on the servers - "see" the same thing. It is like fuzzing on a huge server. - * in intervals of 1/10th of the overall expected runtime of the fuzzing you - sync. This tries a bit to combine both. have some individuality of the - paths each campaign on a server explores, on the other hand if one - gets stuck where another found progress this is handed over making it - unstuck. - -The syncing process itself is very simple. -As the `-M main-$HOSTNAME` instance syncs to all `-S` secondaries as well -as to other fuzzers, you have to copy only this directory to the other -machines. - -Lets say all servers have the `-o out` directory in /target/foo/out, and -you created a file `servers.txt` which contains the hostnames of all -participating servers, plus you have an ssh key deployed to all of them, -then run: +* never: sounds weird, but this makes every server an island and has the chance + the each follow different paths into the target. You can make this even more + interesting by even giving different seeds to each server. +* regularly (~4h): this ensures that all fuzzing campaigns on the servers "see" + the same thing. It is like fuzzing on a huge server. +* in intervals of 1/10th of the overall expected runtime of the fuzzing you + sync. This tries a bit to combine both. have some individuality of the paths + each campaign on a server explores, on the other hand if one gets stuck where + another found progress this is handed over making it unstuck. + +The syncing process itself is very simple. As the `-M main-$HOSTNAME` instance +syncs to all `-S` secondaries as well as to other fuzzers, you have to copy only +this directory to the other machines. + +Lets say all servers have the `-o out` directory in /target/foo/out, and you +created a file `servers.txt` which contains the hostnames of all participating +servers, plus you have an ssh key deployed to all of them, then run: + ```bash for FROM in `cat servers.txt`; do for TO in `cat servers.txt`; do @@ -525,49 +526,52 @@ for FROM in `cat servers.txt`; do done done ``` -You can run this manually, per cron job - as you need it. -There is a more complex and configurable script in `utils/distributed_fuzzing`. + +You can run this manually, per cron job - as you need it. There is a more +complex and configurable script in `utils/distributed_fuzzing`. #### d) The status of the fuzz campaign AFL++ comes with the `afl-whatsup` script to show the status of the fuzzing campaign. -Just supply the directory that afl-fuzz is given with the -o option and -you will see a detailed status of every fuzzer in that campaign plus -a summary. +Just supply the directory that afl-fuzz is given with the -o option and you will +see a detailed status of every fuzzer in that campaign plus a summary. -To have only the summary use the `-s` switch e.g.: `afl-whatsup -s out/` +To have only the summary, use the `-s` switch, e.g. `afl-whatsup -s out/`. -If you have multiple servers then use the command after a sync, or you have -to execute this script per server. +If you have multiple servers, then use the command after a sync or you have to +execute this script per server. -Another tool to inspect the current state and history of a specific instance -is afl-plot, which generates an index.html file and a graphs that show how -the fuzzing instance is performing. -The syntax is `afl-plot instance_dir web_dir`, e.g. `afl-plot out/default /srv/www/htdocs/plot` +Another tool to inspect the current state and history of a specific instance is +afl-plot, which generates an index.html file and a graphs that show how the +fuzzing instance is performing. The syntax is `afl-plot instance_dir web_dir`, +e.g. `afl-plot out/default /srv/www/htdocs/plot`. #### e) Stopping fuzzing, restarting fuzzing, adding new seeds To stop an afl-fuzz run, simply press Control-C. -To restart an afl-fuzz run, just reuse the same command line but replace the -`-i directory` with `-i -` or set `AFL_AUTORESUME=1`. +To restart an afl-fuzz run, just reuse the same command line but replace the `-i +directory` with `-i -` or set `AFL_AUTORESUME=1`. If you want to add new seeds to a fuzzing campaign you can run a temporary -fuzzing instance, e.g. when your main fuzzer is using `-o out` and the new -seeds are in `newseeds/` directory: +fuzzing instance, e.g. when your main fuzzer is using `-o out` and the new seeds +are in `newseeds/` directory: + ``` AFL_BENCH_JUST_ONE=1 AFL_FAST_CAL=1 afl-fuzz -i newseeds -o out -S newseeds -- ./target ``` #### f) Checking the coverage of the fuzzing -The `paths found` value is a bad indicator for checking how good the coverage is. +The `paths found` value is a bad indicator for checking how good the coverage +is. A better indicator - if you use default llvm instrumentation with at least -version 9 - is to use `afl-showmap` with the collect coverage option `-C` on -the output directory: +version 9 - is to use `afl-showmap` with the collect coverage option `-C` on the +output directory: + ``` $ afl-showmap -C -i out -o /dev/null -- ./target -params @@ ... @@ -578,53 +582,67 @@ $ afl-showmap -C -i out -o /dev/null -- ./target -params @@ l'. [+] A coverage of 4331 edges were achieved out of 9960 existing (43.48%) with 7849 input files. ``` + It is even better to check out the exact lines of code that have been reached - and which have not been found so far. -An "easy" helper script for this is [https://github.com/vanhauser-thc/afl-cov](https://github.com/vanhauser-thc/afl-cov), +An "easy" helper script for this is +[https://github.com/vanhauser-thc/afl-cov](https://github.com/vanhauser-thc/afl-cov), just follow the README of that separate project. If you see that an important area or a feature has not been covered so far then try to find an input that is able to reach that and start a new secondary in that fuzzing campaign with that seed as input, let it run for a few minutes, then terminate it. The main node will pick it up and make it available to the -other secondary nodes over time. Set `export AFL_NO_AFFINITY=1` or -`export AFL_TRY_AFFINITY=1` if you have no free core. +other secondary nodes over time. Set `export AFL_NO_AFFINITY=1` or `export +AFL_TRY_AFFINITY=1` if you have no free core. Note that in nearly all cases you can never reach full coverage. A lot of -functionality is usually dependent on exclusive options that would need individual -fuzzing campaigns each with one of these options set. E.g. if you fuzz a library to -convert image formats and your target is the png to tiff API then you will not -touch any of the other library APIs and features. +functionality is usually dependent on exclusive options that would need +individual fuzzing campaigns each with one of these options set. E.g. if you +fuzz a library to convert image formats and your target is the png to tiff API +then you will not touch any of the other library APIs and features. #### g) How long to fuzz a target? -This is a difficult question. -Basically if no new path is found for a long time (e.g. for a day or a week) -then you can expect that your fuzzing won't be fruitful anymore. -However often this just means that you should switch out secondaries for -others, e.g. custom mutator modules, sync to very different fuzzers, etc. +This is a difficult question. Basically if no new path is found for a long time +(e.g. for a day or a week) then you can expect that your fuzzing won't be +fruitful anymore. However, often this just means that you should switch out +secondaries for others, e.g. custom mutator modules, sync to very different +fuzzers, etc. Keep the queue/ directory (for future fuzzings of the same or similar targets) -and use them to seed other good fuzzers like libfuzzer with the -entropic -switch or honggfuzz. +and use them to seed other good fuzzers like libfuzzer with the -entropic switch +or honggfuzz. #### h) Improve the speed! - * Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 speed increase) - * If you do not use shmem persistent mode, use `AFL_TMPDIR` to point the input file on a tempfs location, see [env_variables.md](env_variables.md) - * Linux: Improve kernel performance: modify `/etc/default/grub`, set `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then `update-grub` and `reboot` (warning: makes the system more insecure) - you can also just run `sudo afl-persistent-config` - * Linux: Running on an `ext2` filesystem with `noatime` mount option will be a bit faster than on any other journaling filesystem - * Use your cores! [b) Using multiple cores](#b-using-multiple-cores) - * Run `sudo afl-system-config` before starting the first afl-fuzz instance after a reboot +* Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 + speed increase) +* If you do not use shmem persistent mode, use `AFL_TMPDIR` to point the input + file on a tempfs location, see [env_variables.md](env_variables.md) +* Linux: Improve kernel performance: modify `/etc/default/grub`, set + `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off + mitigations=off no_stf_barrier noibpb noibrs nopcid nopti + nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off + spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then + `update-grub` and `reboot` (warning: makes the system more insecure) - you can + also just run `sudo afl-persistent-config` +* Linux: Running on an `ext2` filesystem with `noatime` mount option will be a + bit faster than on any other journaling filesystem +* Use your cores! [b) Using multiple cores](#b-using-multiple-cores) +* Run `sudo afl-system-config` before starting the first afl-fuzz instance after + a reboot ### The End -Check out the [FAQ](FAQ.md) if it maybe answers your question (that -you might not even have known you had ;-) ). +Check out the [FAQ](FAQ.md) if it maybe answers your question (that you might +not even have known you had ;-) ). This is basically all you need to know to professionally run fuzzing campaigns. -If you want to know more, the tons of texts in [docs/](./) will have you covered. +If you want to know more, the tons of texts in [docs/](./) will have you +covered. Note that there are also a lot of tools out there that help fuzzing with AFL++ -(some might be deprecated or unsupported), see [third_party_tools.md](third_party_tools.md). \ No newline at end of file +(some might be deprecated or unsupported), see +[third_party_tools.md](third_party_tools.md). \ No newline at end of file diff --git a/docs/important_changes.md b/docs/important_changes.md index 0c5c2243..877dfab2 100644 --- a/docs/important_changes.md +++ b/docs/important_changes.md @@ -36,7 +36,7 @@ behaviours and defaults: shared libraries, etc. Additionally QEMU 5.1 supports more CPU targets so this is really worth it. * When instrumenting targets, afl-cc will not supersede optimizations anymore - if any were given. This allows to fuzz targets build regularly like those + if any were given. This allows to fuzz targets build regularly like those for debug or release versions. * afl-fuzz: * if neither -M or -S is specified, `-S default` is assumed, so more @@ -47,7 +47,7 @@ behaviours and defaults: * -m none is now default, set memory limits (in MB) with e.g. -m 250 * deterministic fuzzing is now disabled by default (unless using -M) and can be enabled with -D - * a caching of testcases can now be performed and can be modified by + * a caching of test cases can now be performed and can be modified by editing config.h for TESTCASE_CACHE or by specifying the env variable `AFL_TESTCACHE_SIZE` (in MB). Good values are between 50-500 (default: 50). * -M mains do not perform trimming diff --git a/docs/interpreting_output.md b/docs/interpreting_output.md deleted file mode 100644 index 4bd705f2..00000000 --- a/docs/interpreting_output.md +++ /dev/null @@ -1,71 +0,0 @@ -# Interpreting output - -See the [status_screen.md](status_screen.md) file for information on -how to interpret the displayed stats and monitor the health of the process. Be -sure to consult this file especially if any UI elements are highlighted in red. - -The fuzzing process will continue until you press Ctrl-C. At a minimum, you want -to allow the fuzzer to complete one queue cycle, which may take anywhere from a -couple of hours to a week or so. - -There are three subdirectories created within the output directory and updated -in real-time: - - - queue/ - test cases for every distinctive execution path, plus all the - starting files given by the user. This is the synthesized corpus - mentioned in section 2. - - Before using this corpus for any other purposes, you can shrink - it to a smaller size using the afl-cmin tool. The tool will find - a smaller subset of files offering equivalent edge coverage. - - - crashes/ - unique test cases that cause the tested program to receive a - fatal signal (e.g., SIGSEGV, SIGILL, SIGABRT). The entries are - grouped by the received signal. - - - hangs/ - unique test cases that cause the tested program to time out. The - default time limit before something is classified as a hang is - the larger of 1 second and the value of the -t parameter. - The value can be fine-tuned by setting AFL_HANG_TMOUT, but this - is rarely necessary. - -Crashes and hangs are considered "unique" if the associated execution paths -involve any state transitions not seen in previously-recorded faults. If a -single bug can be reached in multiple ways, there will be some count inflation -early in the process, but this should quickly taper off. - -The file names for crashes and hangs are correlated with the parent, non-faulting -queue entries. This should help with debugging. - -When you can't reproduce a crash found by afl-fuzz, the most likely cause is -that you are not setting the same memory limit as used by the tool. Try: - -```shell -LIMIT_MB=50 -( ulimit -Sv $[LIMIT_MB << 10]; /path/to/tested_binary ... ) -``` - -Change LIMIT_MB to match the -m parameter passed to afl-fuzz. On OpenBSD, -also change -Sv to -Sd. - -Any existing output directory can be also used to resume aborted jobs; try: - -```shell -./afl-fuzz -i- -o existing_output_dir [...etc...] -``` - -If you have gnuplot installed, you can also generate some pretty graphs for any -active fuzzing task using afl-plot. For an example of how this looks like, -see [https://lcamtuf.coredump.cx/afl/plot/](https://lcamtuf.coredump.cx/afl/plot/). - -You can also manually build and install afl-plot-ui, which is a helper utility -for showing the graphs generated by afl-plot in a graphical window using GTK. -You can build and install it as follows - -```shell -sudo apt install libgtk-3-0 libgtk-3-dev pkg-config -cd utils/plot_ui -make -cd ../../ -sudo make install -``` diff --git a/docs/status_screen.md b/docs/status_screen.md deleted file mode 100644 index b1cb9696..00000000 --- a/docs/status_screen.md +++ /dev/null @@ -1,444 +0,0 @@ -# Understanding the status screen - -This document provides an overview of the status screen - plus tips for -troubleshooting any warnings and red text shown in the UI. See README.md for -the general instruction manual. - -## A note about colors - -The status screen and error messages use colors to keep things readable and -attract your attention to the most important details. For example, red almost -always means "consult this doc" :-) - -Unfortunately, the UI will render correctly only if your terminal is using -traditional un*x palette (white text on black background) or something close -to that. - -If you are using inverse video, you may want to change your settings, say: - -- For GNOME Terminal, go to `Edit > Profile` preferences, select the "colors" tab, and from the list of built-in schemes, choose "white on black". -- For the MacOS X Terminal app, open a new window using the "Pro" scheme via the `Shell > New Window` menu (or make "Pro" your default). - -Alternatively, if you really like your current colors, you can edit config.h -to comment out USE_COLORS, then do `make clean all`. - -I'm not aware of any other simple way to make this work without causing -other side effects - sorry about that. - -With that out of the way, let's talk about what's actually on the screen... - -### The status bar - -``` -american fuzzy lop ++3.01a (default) [fast] {0} -``` - -The top line shows you which mode afl-fuzz is running in -(normal: "american fuzy lop", crash exploration mode: "peruvian rabbit mode") -and the version of AFL++. -Next to the version is the banner, which, if not set with -T by hand, will -either show the binary name being fuzzed, or the -M/-S main/secondary name for -parallel fuzzing. -Second to last is the power schedule mode being run (default: fast). -Finally, the last item is the CPU id. - -### Process timing - -``` - +----------------------------------------------------+ - | run time : 0 days, 8 hrs, 32 min, 43 sec | - | last new path : 0 days, 0 hrs, 6 min, 40 sec | - | last uniq crash : none seen yet | - | last uniq hang : 0 days, 1 hrs, 24 min, 32 sec | - +----------------------------------------------------+ -``` - -This section is fairly self-explanatory: it tells you how long the fuzzer has -been running and how much time has elapsed since its most recent finds. This is -broken down into "paths" (a shorthand for test cases that trigger new execution -patterns), crashes, and hangs. - -When it comes to timing: there is no hard rule, but most fuzzing jobs should be -expected to run for days or weeks; in fact, for a moderately complex project, the -first pass will probably take a day or so. Every now and then, some jobs -will be allowed to run for months. - -There's one important thing to watch out for: if the tool is not finding new -paths within several minutes of starting, you're probably not invoking the -target binary correctly and it never gets to parse the input files we're -throwing at it; another possible explanations are that the default memory limit -(`-m`) is too restrictive, and the program exits after failing to allocate a -buffer very early on; or that the input files are patently invalid and always -fail a basic header check. - -If there are no new paths showing up for a while, you will eventually see a big -red warning in this section, too :-) - -### Overall results - -``` - +-----------------------+ - | cycles done : 0 | - | total paths : 2095 | - | uniq crashes : 0 | - | uniq hangs : 19 | - +-----------------------+ -``` - -The first field in this section gives you the count of queue passes done so far - that is, the number of times the fuzzer went over all the interesting test -cases discovered so far, fuzzed them, and looped back to the very beginning. -Every fuzzing session should be allowed to complete at least one cycle; and -ideally, should run much longer than that. - -As noted earlier, the first pass can take a day or longer, so sit back and -relax. - -To help make the call on when to hit `Ctrl-C`, the cycle counter is color-coded. -It is shown in magenta during the first pass, progresses to yellow if new finds -are still being made in subsequent rounds, then blue when that ends - and -finally, turns green after the fuzzer hasn't been seeing any action for a -longer while. - -The remaining fields in this part of the screen should be pretty obvious: -there's the number of test cases ("paths") discovered so far, and the number of -unique faults. The test cases, crashes, and hangs can be explored in real-time -by browsing the output directory, as discussed in README.md. - -### Cycle progress - -``` - +-------------------------------------+ - | now processing : 1296 (61.86%) | - | paths timed out : 0 (0.00%) | - +-------------------------------------+ -``` - -This box tells you how far along the fuzzer is with the current queue cycle: it -shows the ID of the test case it is currently working on, plus the number of -inputs it decided to ditch because they were persistently timing out. - -The "*" suffix sometimes shown in the first line means that the currently -processed path is not "favored" (a property discussed later on). - -### Map coverage - -``` - +--------------------------------------+ - | map density : 10.15% / 29.07% | - | count coverage : 4.03 bits/tuple | - +--------------------------------------+ -``` - -The section provides some trivia about the coverage observed by the -instrumentation embedded in the target binary. - -The first line in the box tells you how many branch tuples we have already -hit, in proportion to how much the bitmap can hold. The number on the left -describes the current input; the one on the right is the value for the entire -input corpus. - -Be wary of extremes: - - - Absolute numbers below 200 or so suggest one of three things: that the - program is extremely simple; that it is not instrumented properly (e.g., - due to being linked against a non-instrumented copy of the target - library); or that it is bailing out prematurely on your input test cases. - The fuzzer will try to mark this in pink, just to make you aware. - - Percentages over 70% may very rarely happen with very complex programs - that make heavy use of template-generated code. - Because high bitmap density makes it harder for the fuzzer to reliably - discern new program states, I recommend recompiling the binary with - `AFL_INST_RATIO=10` or so and trying again (see env_variables.md). - The fuzzer will flag high percentages in red. Chances are, you will never - see that unless you're fuzzing extremely hairy software (say, v8, perl, - ffmpeg). - -The other line deals with the variability in tuple hit counts seen in the -binary. In essence, if every taken branch is always taken a fixed number of -times for all the inputs we have tried, this will read `1.00`. As we manage -to trigger other hit counts for every branch, the needle will start to move -toward `8.00` (every bit in the 8-bit map hit), but will probably never -reach that extreme. - -Together, the values can be useful for comparing the coverage of several -different fuzzing jobs that rely on the same instrumented binary. - -### Stage progress - -``` - +-------------------------------------+ - | now trying : interest 32/8 | - | stage execs : 3996/34.4k (11.62%) | - | total execs : 27.4M | - | exec speed : 891.7/sec | - +-------------------------------------+ -``` - -This part gives you an in-depth peek at what the fuzzer is actually doing right -now. It tells you about the current stage, which can be any of: - - - calibration - a pre-fuzzing stage where the execution path is examined - to detect anomalies, establish baseline execution speed, and so on. Executed - very briefly whenever a new find is being made. - - trim L/S - another pre-fuzzing stage where the test case is trimmed to the - shortest form that still produces the same execution path. The length (L) - and stepover (S) are chosen in general relationship to file size. - - bitflip L/S - deterministic bit flips. There are L bits toggled at any given - time, walking the input file with S-bit increments. The current L/S variants - are: `1/1`, `2/1`, `4/1`, `8/8`, `16/8`, `32/8`. - - arith L/8 - deterministic arithmetics. The fuzzer tries to subtract or add - small integers to 8-, 16-, and 32-bit values. The stepover is always 8 bits. - - interest L/8 - deterministic value overwrite. The fuzzer has a list of known - "interesting" 8-, 16-, and 32-bit values to try. The stepover is 8 bits. - - extras - deterministic injection of dictionary terms. This can be shown as - "user" or "auto", depending on whether the fuzzer is using a user-supplied - dictionary (`-x`) or an auto-created one. You will also see "over" or "insert", - depending on whether the dictionary words overwrite existing data or are - inserted by offsetting the remaining data to accommodate their length. - - havoc - a sort-of-fixed-length cycle with stacked random tweaks. The - operations attempted during this stage include bit flips, overwrites with - random and "interesting" integers, block deletion, block duplication, plus - assorted dictionary-related operations (if a dictionary is supplied in the - first place). - - splice - a last-resort strategy that kicks in after the first full queue - cycle with no new paths. It is equivalent to 'havoc', except that it first - splices together two random inputs from the queue at some arbitrarily - selected midpoint. - - sync - a stage used only when `-M` or `-S` is set (see parallel_fuzzing.md). - No real fuzzing is involved, but the tool scans the output from other - fuzzers and imports test cases as necessary. The first time this is done, - it may take several minutes or so. - -The remaining fields should be fairly self-evident: there's the exec count -progress indicator for the current stage, a global exec counter, and a -benchmark for the current program execution speed. This may fluctuate from -one test case to another, but the benchmark should be ideally over 500 execs/sec -most of the time - and if it stays below 100, the job will probably take very -long. - -The fuzzer will explicitly warn you about slow targets, too. If this happens, -see the [perf_tips.md](perf_tips.md) file included with the fuzzer for ideas on how to speed -things up. - -### Findings in depth - -``` - +--------------------------------------+ - | favored paths : 879 (41.96%) | - | new edges on : 423 (20.19%) | - | total crashes : 0 (0 unique) | - | total tmouts : 24 (19 unique) | - +--------------------------------------+ -``` - -This gives you several metrics that are of interest mostly to complete nerds. -The section includes the number of paths that the fuzzer likes the most based -on a minimization algorithm baked into the code (these will get considerably -more air time), and the number of test cases that actually resulted in better -edge coverage (versus just pushing the branch hit counters up). There are also -additional, more detailed counters for crashes and timeouts. - -Note that the timeout counter is somewhat different from the hang counter; this -one includes all test cases that exceeded the timeout, even if they did not -exceed it by a margin sufficient to be classified as hangs. - -### Fuzzing strategy yields - -``` - +-----------------------------------------------------+ - | bit flips : 57/289k, 18/289k, 18/288k | - | byte flips : 0/36.2k, 4/35.7k, 7/34.6k | - | arithmetics : 53/2.54M, 0/537k, 0/55.2k | - | known ints : 8/322k, 12/1.32M, 10/1.70M | - | dictionary : 9/52k, 1/53k, 1/24k | - |havoc/splice : 1903/20.0M, 0/0 | - |py/custom/rq : unused, 53/2.54M, unused | - | trim/eff : 20.31%/9201, 17.05% | - +-----------------------------------------------------+ -``` - -This is just another nerd-targeted section keeping track of how many paths we -have netted, in proportion to the number of execs attempted, for each of the -fuzzing strategies discussed earlier on. This serves to convincingly validate -assumptions about the usefulness of the various approaches taken by afl-fuzz. - -The trim strategy stats in this section are a bit different than the rest. -The first number in this line shows the ratio of bytes removed from the input -files; the second one corresponds to the number of execs needed to achieve this -goal. Finally, the third number shows the proportion of bytes that, although -not possible to remove, were deemed to have no effect and were excluded from -some of the more expensive deterministic fuzzing steps. - -Note that when deterministic mutation mode is off (which is the default -because it is not very efficient) the first five lines display -"disabled (default, enable with -D)". - -Only what is activated will have counter shown. - -### Path geometry - -``` - +---------------------+ - | levels : 5 | - | pending : 1570 | - | pend fav : 583 | - | own finds : 0 | - | imported : 0 | - | stability : 100.00% | - +---------------------+ -``` - -The first field in this section tracks the path depth reached through the -guided fuzzing process. In essence: the initial test cases supplied by the -user are considered "level 1". The test cases that can be derived from that -through traditional fuzzing are considered "level 2"; the ones derived by -using these as inputs to subsequent fuzzing rounds are "level 3"; and so forth. -The maximum depth is therefore a rough proxy for how much value you're getting -out of the instrumentation-guided approach taken by afl-fuzz. - -The next field shows you the number of inputs that have not gone through any -fuzzing yet. The same stat is also given for "favored" entries that the fuzzer -really wants to get to in this queue cycle (the non-favored entries may have to -wait a couple of cycles to get their chance). - -Next, we have the number of new paths found during this fuzzing section and -imported from other fuzzer instances when doing parallelized fuzzing; and the -extent to which identical inputs appear to sometimes produce variable behavior -in the tested binary. - -That last bit is actually fairly interesting: it measures the consistency of -observed traces. If a program always behaves the same for the same input data, -it will earn a score of 100%. When the value is lower but still shown in purple, -the fuzzing process is unlikely to be negatively affected. If it goes into red, -you may be in trouble, since AFL will have difficulty discerning between -meaningful and "phantom" effects of tweaking the input file. - -Now, most targets will just get a 100% score, but when you see lower figures, -there are several things to look at: - - - The use of uninitialized memory in conjunction with some intrinsic sources - of entropy in the tested binary. Harmless to AFL, but could be indicative - of a security bug. - - Attempts to manipulate persistent resources, such as left over temporary - files or shared memory objects. This is usually harmless, but you may want - to double-check to make sure the program isn't bailing out prematurely. - Running out of disk space, SHM handles, or other global resources can - trigger this, too. - - Hitting some functionality that is actually designed to behave randomly. - Generally harmless. For example, when fuzzing sqlite, an input like - `select random();` will trigger a variable execution path. - - Multiple threads executing at once in semi-random order. This is harmless - when the 'stability' metric stays over 90% or so, but can become an issue - if not. Here's what to try: - * Use afl-clang-fast from [instrumentation](../instrumentation/) - it uses a thread-local tracking - model that is less prone to concurrency issues, - * See if the target can be compiled or run without threads. Common - `./configure` options include `--without-threads`, `--disable-pthreads`, or - `--disable-openmp`. - * Replace pthreads with GNU Pth (https://www.gnu.org/software/pth/), which - allows you to use a deterministic scheduler. - - In persistent mode, minor drops in the "stability" metric can be normal, - because not all the code behaves identically when re-entered; but major - dips may signify that the code within `__AFL_LOOP()` is not behaving - correctly on subsequent iterations (e.g., due to incomplete clean-up or - reinitialization of the state) and that most of the fuzzing effort goes - to waste. - -The paths where variable behavior is detected are marked with a matching entry -in the `/queue/.state/variable_behavior/` directory, so you can look -them up easily. - -### CPU load - -``` - [cpu: 25%] -``` - -This tiny widget shows the apparent CPU utilization on the local system. It is -calculated by taking the number of processes in the "runnable" state, and then -comparing it to the number of logical cores on the system. - -If the value is shown in green, you are using fewer CPU cores than available on -your system and can probably parallelize to improve performance; for tips on -how to do that, see parallel_fuzzing.md. - -If the value is shown in red, your CPU is *possibly* oversubscribed, and -running additional fuzzers may not give you any benefits. - -Of course, this benchmark is very simplistic; it tells you how many processes -are ready to run, but not how resource-hungry they may be. It also doesn't -distinguish between physical cores, logical cores, and virtualized CPUs; the -performance characteristics of each of these will differ quite a bit. - -If you want a more accurate measurement, you can run the `afl-gotcpu` utility from the command line. - -### Addendum: status and plot files - -For unattended operation, some of the key status screen information can be also -found in a machine-readable format in the fuzzer_stats file in the output -directory. This includes: - - - `start_time` - unix time indicating the start time of afl-fuzz - - `last_update` - unix time corresponding to the last update of this file - - `run_time` - run time in seconds to the last update of this file - - `fuzzer_pid` - PID of the fuzzer process - - `cycles_done` - queue cycles completed so far - - `cycles_wo_finds` - number of cycles without any new paths found - - `execs_done` - number of execve() calls attempted - - `execs_per_sec` - overall number of execs per second - - `paths_total` - total number of entries in the queue - - `paths_favored` - number of queue entries that are favored - - `paths_found` - number of entries discovered through local fuzzing - - `paths_imported` - number of entries imported from other instances - - `max_depth` - number of levels in the generated data set - - `cur_path` - currently processed entry number - - `pending_favs` - number of favored entries still waiting to be fuzzed - - `pending_total` - number of all entries waiting to be fuzzed - - `variable_paths` - number of test cases showing variable behavior - - `stability` - percentage of bitmap bytes that behave consistently - - `bitmap_cvg` - percentage of edge coverage found in the map so far - - `unique_crashes` - number of unique crashes recorded - - `unique_hangs` - number of unique hangs encountered - - `last_path` - seconds since the last path was found - - `last_crash` - seconds since the last crash was found - - `last_hang` - seconds since the last hang was found - - `execs_since_crash` - execs since the last crash was found - - `exec_timeout` - the -t command line value - - `slowest_exec_ms` - real time of the slowest execution in ms - - `peak_rss_mb` - max rss usage reached during fuzzing in MB - - `edges_found` - how many edges have been found - - `var_byte_count` - how many edges are non-deterministic - - `afl_banner` - banner text (e.g. the target name) - - `afl_version` - the version of AFL used - - `target_mode` - default, persistent, qemu, unicorn, non-instrumented - - `command_line` - full command line used for the fuzzing session - -Most of these map directly to the UI elements discussed earlier on. - -On top of that, you can also find an entry called `plot_data`, containing a -plottable history for most of these fields. If you have gnuplot installed, you -can turn this into a nice progress report with the included `afl-plot` tool. - - -### Addendum: Automatically send metrics with StatsD - -In a CI environment or when running multiple fuzzers, it can be tedious to -log into each of them or deploy scripts to read the fuzzer statistics. -Using `AFL_STATSD` (and the other related environment variables `AFL_STATSD_HOST`, -`AFL_STATSD_PORT`, `AFL_STATSD_TAGS_FLAVOR`) you can automatically send metrics -to your favorite StatsD server. Depending on your StatsD server you will be able -to monitor, trigger alerts or perform actions based on these metrics (e.g: alert on -slow exec/s for a new build, threshold of crashes, time since last crash > X, etc). - -The selected metrics are a subset of all the metrics found in the status and in -the plot file. The list is the following: `cycle_done`, `cycles_wo_finds`, -`execs_done`,`execs_per_sec`, `paths_total`, `paths_favored`, `paths_found`, -`paths_imported`, `max_depth`, `cur_path`, `pending_favs`, `pending_total`, -`variable_paths`, `unique_crashes`, `unique_hangs`, `total_crashes`, -`slowest_exec_ms`, `edges_found`, `var_byte_count`, `havoc_expansion`. -Their definitions can be found in the addendum above. - -When using multiple fuzzer instances with StatsD it is *strongly* recommended to setup -the flavor (AFL_STATSD_TAGS_FLAVOR) to match your StatsD server. This will allow you -to see individual fuzzer performance, detect bad ones, see the progress of each -strategy... diff --git a/docs/third_party_tools.md b/docs/third_party_tools.md index ba96d0ce..446d373c 100644 --- a/docs/third_party_tools.md +++ b/docs/third_party_tools.md @@ -1,12 +1,12 @@ # Tools that help fuzzing with AFL++ Speeding up fuzzing: - * [libfiowrapper](https://github.com/marekzmyslowski/libfiowrapper) - if the function you want to fuzz requires loading a file, this allows using the shared memory testcase feature :-) - recommended. + * [libfiowrapper](https://github.com/marekzmyslowski/libfiowrapper) - if the function you want to fuzz requires loading a file, this allows using the shared memory test case feature :-) - recommended. Minimization of test cases: * [afl-pytmin](https://github.com/ilsani/afl-pytmin) - a wrapper for afl-tmin that tries to speed up the process of minimization of a single test case by using many CPU cores. - * [afl-ddmin-mod](https://github.com/MarkusTeufelberger/afl-ddmin-mod) - a variation of afl-tmin based on the ddmin algorithm. - * [halfempty](https://github.com/googleprojectzero/halfempty) - is a fast utility for minimizing test cases by Tavis Ormandy based on parallelization. + * [afl-ddmin-mod](https://github.com/MarkusTeufelberger/afl-ddmin-mod) - a variation of afl-tmin based on the ddmin algorithm. + * [halfempty](https://github.com/googleprojectzero/halfempty) - is a fast utility for minimizing test cases by Tavis Ormandy based on parallelization. Distributed execution: * [disfuzz-afl](https://github.com/MartijnB/disfuzz-afl) - distributed fuzzing for AFL. diff --git a/qemu_mode/libqasan/README.md b/qemu_mode/libqasan/README.md index 4a241233..6a65c12b 100644 --- a/qemu_mode/libqasan/README.md +++ b/qemu_mode/libqasan/README.md @@ -19,7 +19,7 @@ finding capabilities during fuzzing) is WIP. ### When should I use QASan? If your target binary is PIC x86_64, you should also give a try to -[retrowrite](https://github.com/HexHive/retrowrite) for static rewriting. +[RetroWrite](https://github.com/HexHive/retrowrite) for static rewriting. If it fails, or if your binary is for another architecture, or you want to use persistent and snapshot mode, AFL++ QASan mode is what you want/have to use. diff --git a/unicorn_mode/samples/persistent/COMPILE.md b/unicorn_mode/samples/persistent/COMPILE.md index 111dfc54..9f2ae718 100644 --- a/unicorn_mode/samples/persistent/COMPILE.md +++ b/unicorn_mode/samples/persistent/COMPILE.md @@ -1,13 +1,16 @@ # C Sample This shows a simple persistent harness for unicornafl in C. -In contrast to the normal c harness, this harness manually resets the unicorn state on each new input. -Thanks to this, we can rerun the testcase in unicorn multiple times, without the need to fork again. +In contrast to the normal c harness, this harness manually resets the unicorn +state on each new input. +Thanks to this, we can rerun the test case in unicorn multiple times, without +the need to fork again. ## Compiling sample.c The target can be built using the `make` command. Just make sure you have built unicorn support first: + ```bash cd /path/to/afl/unicorn_mode ./build_unicorn_support.sh @@ -19,6 +22,7 @@ You don't need to compile persistent_target.c since a X86_64 binary version is pre-built and shipped in this sample folder. This file documents how the binary was built in case you want to rebuild it or recompile it for any reason. -The pre-built binary (persistent_target_x86_64.bin) was built using -g -O0 in gcc. +The pre-built binary (persistent_target_x86_64.bin) was built using -g -O0 in +gcc. -We then load the binary and we execute the main function directly. +We then load the binary and we execute the main function directly. \ No newline at end of file diff --git a/utils/aflpp_driver/README.md b/utils/aflpp_driver/README.md index 30e2412f..4560be2b 100644 --- a/utils/aflpp_driver/README.md +++ b/utils/aflpp_driver/README.md @@ -7,15 +7,15 @@ targets. Just do `afl-clang-fast++ -o fuzz fuzzer_harness.cc libAFLDriver.a [plus required linking]`. -You can also sneakily do this little trick: +You can also sneakily do this little trick: If this is the clang compile command to build for libfuzzer: `clang++ -o fuzz -fsanitize=fuzzer fuzzer_harness.cc -lfoo` then just switch `clang++` with `afl-clang-fast++` and our compiler will magically insert libAFLDriver.a :) -To use shared-memory testcases, you need nothing to do. -To use stdin testcases give `-` as the only command line parameter. -To use file input testcases give `@@` as the only command line parameter. +To use shared-memory test cases, you need nothing to do. +To use stdin test cases, give `-` as the only command line parameter. +To use file input test cases, give `@@` as the only command line parameter. IMPORTANT: if you use `afl-cmin` or `afl-cmin.bash` then either pass `-` or `@@` as command line parameters. @@ -30,8 +30,8 @@ are to be fuzzed in qemu_mode. So we compile them with clang/clang++, without `clang++ -o fuzz fuzzer_harness.cc libAFLQemuDriver.a [plus required linking]`. - Then just do (where the name of the binary is `fuzz`): + ``` AFL_QEMU_PERSISTENT_ADDR=0x$(nm fuzz | grep "T LLVMFuzzerTestOneInput" | awk '{print $1}') AFL_QEMU_PERSISTENT_HOOK=/path/to/aflpp_qemu_driver_hook.so afl-fuzz -Q ... -- ./fuzz` @@ -40,4 +40,4 @@ AFL_QEMU_PERSISTENT_HOOK=/path/to/aflpp_qemu_driver_hook.so afl-fuzz -Q ... -- . if you use afl-cmin or `afl-showmap -C` with the aflpp_qemu_driver you need to set the set same AFL_QEMU_... (or AFL_FRIDA_...) environment variables. If you want to use afl-showmap (without -C) or afl-cmin.bash then you may not -set these environment variables and rather set `AFL_QEMU_DRIVER_NO_HOOK=1`. +set these environment variables and rather set `AFL_QEMU_DRIVER_NO_HOOK=1`. \ No newline at end of file -- cgit 1.4.1 From 22726315c3bd31f53c2f4bcf1f8649767ec5276a Mon Sep 17 00:00:00 2001 From: llzmb <46303940+llzmb@users.noreply.github.com> Date: Wed, 24 Nov 2021 08:11:15 +0100 Subject: Merge various files into "fuzzing_in_depth.md" --- docs/beyond_crashes.md | 23 ------- docs/choosing_testcases.md | 19 ------ docs/fuzzing_in_depth.md | 151 ++++++++++++++++++++++++++++++++++++++++----- docs/limitations.md | 37 ----------- docs/triaging_crashes.md | 46 -------------- 5 files changed, 135 insertions(+), 141 deletions(-) delete mode 100644 docs/beyond_crashes.md delete mode 100644 docs/choosing_testcases.md delete mode 100644 docs/limitations.md delete mode 100644 docs/triaging_crashes.md (limited to 'docs/fuzzing_in_depth.md') diff --git a/docs/beyond_crashes.md b/docs/beyond_crashes.md deleted file mode 100644 index 4836419c..00000000 --- a/docs/beyond_crashes.md +++ /dev/null @@ -1,23 +0,0 @@ -# Going beyond crashes - -Fuzzing is a wonderful and underutilized technique for discovering non-crashing -design and implementation errors, too. Quite a few interesting bugs have been -found by modifying the target programs to call abort() when say: - - - Two bignum libraries produce different outputs when given the same - fuzzer-generated input, - - - An image library produces different outputs when asked to decode the same - input image several times in a row, - - - A serialization / deserialization library fails to produce stable outputs - when iteratively serializing and deserializing fuzzer-supplied data, - - - A compression library produces an output inconsistent with the input file - when asked to compress and then decompress a particular blob. - -Implementing these or similar sanity checks usually takes very little time; -if you are the maintainer of a particular package, you can make this code -conditional with `#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` (a flag also -shared with libfuzzer and honggfuzz) or `#ifdef __AFL_COMPILER` (this one is -just for AFL). \ No newline at end of file diff --git a/docs/choosing_testcases.md b/docs/choosing_testcases.md deleted file mode 100644 index 25002929..00000000 --- a/docs/choosing_testcases.md +++ /dev/null @@ -1,19 +0,0 @@ -# Choosing initial test cases - -To operate correctly, the fuzzer requires one or more starting file that -contains a good example of the input data normally expected by the targeted -application. There are two basic rules: - - - Keep the files small. Under 1 kB is ideal, although not strictly necessary. - For a discussion of why size matters, see [perf_tips.md](perf_tips.md). - - - Use multiple test cases only if they are functionally different from - each other. There is no point in using fifty different vacation photos - to fuzz an image library. - -You can find many good examples of starting files in the testcases/ subdirectory -that comes with this tool. - -PS. If a large corpus of data is available for screening, you may want to use -the afl-cmin utility to identify a subset of functionally distinct files that -exercise different code paths in the target binary. \ No newline at end of file diff --git a/docs/fuzzing_in_depth.md b/docs/fuzzing_in_depth.md index 5b4a9df7..4481bce6 100644 --- a/docs/fuzzing_in_depth.md +++ b/docs/fuzzing_in_depth.md @@ -13,7 +13,7 @@ Fuzzing source code is a three-step process: 3. Perform the fuzzing of the target by randomly mutating input and assessing if a generated input was processed in a new path in the target binary. -### 1. Instrumenting that target +### 1. Instrumenting the target #### a) Selecting the best AFL++ compiler for instrumenting the target @@ -123,7 +123,7 @@ AFL++ performs "never zero" counting in its bitmap. You can read more about this here: * [instrumentation/README.neverzero.md](../instrumentation/README.neverzero.md) -#### c) Sanitizers +#### c) Selecting sanitizers It is possible to use sanitizers when instrumenting targets for fuzzing, which allows you to find bugs that would not necessarily result in a crash. @@ -171,7 +171,7 @@ CFISAN. You might need to experiment which sanitizers you can combine in a target (which means more instances can be run without a sanitized target, which is more effective). -#### d) Modify the target +#### d) Modifying the target If the target has features that make fuzzing more difficult, e.g. checksums, HMAC, etc. then modify the source code so that checks for these values are @@ -188,7 +188,7 @@ products by eliminating these checks within these AFL specific blocks: All AFL++ compilers will set this preprocessor definition automatically. -#### e) Instrument the target +#### e) Instrumenting the target In this step the target source code is compiled so that it can be fuzzed. @@ -272,6 +272,7 @@ it for a hobby and not professionally :-). libfuzzer `LLVMFuzzerTestOneInput()` harnesses are the defacto standard for fuzzing, and they can be used with AFL++ (and honggfuzz) as well! + Compiling them is as simple as: ``` @@ -294,15 +295,25 @@ For more information, see As you fuzz the target with mutated input, having as diverse inputs for the target as possible improves the efficiency a lot. -#### a) Collect inputs +#### a) Collecting inputs + +To operate correctly, the fuzzer requires one or more starting files that +contain a good example of the input data normally expected by the targeted +application. There are two basic rules: + +- Keep the files small. Under 1 kB is ideal, although not strictly necessary. + For a discussion of why size matters, see [perf_tips.md](perf_tips.md). -Try to gather valid inputs for the target from wherever you can. E.g. if it is -the PNG picture format try to find as many png files as possible, e.g. from -reported bugs, test suites, random downloads from the internet, unit test case -data - from all kind of PNG software. +- Use multiple test cases only if they are functionally different from each + other. There is no point in using fifty different vacation photos to fuzz an + image library. -If the input format is not known, you can also modify a target program to write -normal data it receives and processes to a file and use these. +You can find many good examples of starting files in the +[testcases/](../testcases) subdirectory that comes with this tool. + +PS. If a large corpus of data is available for screening, you may want to use +the afl-cmin utility to identify a subset of functionally distinct files that +exercise different code paths in the target binary. #### b) Making the input corpus unique @@ -535,10 +546,10 @@ complex and configurable script in `utils/distributed_fuzzing`. AFL++ comes with the `afl-whatsup` script to show the status of the fuzzing campaign. -Just supply the directory that afl-fuzz is given with the -o option and you will -see a detailed status of every fuzzer in that campaign plus a summary. +Just supply the directory that afl-fuzz is given with the `-o` option and you +will see a detailed status of every fuzzer in that campaign plus a summary. -To have only the summary, use the `-s` switch, e.g. `afl-whatsup -s out/`. +To have only the summary, use the `-s` switch, e.g., `afl-whatsup -s out/`. If you have multiple servers, then use the command after a sync or you have to execute this script per server. @@ -546,7 +557,7 @@ execute this script per server. Another tool to inspect the current state and history of a specific instance is afl-plot, which generates an index.html file and a graphs that show how the fuzzing instance is performing. The syntax is `afl-plot instance_dir web_dir`, -e.g. `afl-plot out/default /srv/www/htdocs/plot`. +e.g., `afl-plot out/default /srv/www/htdocs/plot`. #### e) Stopping fuzzing, restarting fuzzing, adding new seeds @@ -599,7 +610,7 @@ AFL_TRY_AFFINITY=1` if you have no free core. Note that in nearly all cases you can never reach full coverage. A lot of functionality is usually dependent on exclusive options that would need -individual fuzzing campaigns each with one of these options set. E.g. if you +individual fuzzing campaigns each with one of these options set. E.g., if you fuzz a library to convert image formats and your target is the png to tiff API then you will not touch any of the other library APIs and features. @@ -634,6 +645,114 @@ or honggfuzz. * Run `sudo afl-system-config` before starting the first afl-fuzz instance after a reboot +#### i) Going beyond crashes + +Fuzzing is a wonderful and underutilized technique for discovering non-crashing +design and implementation errors, too. Quite a few interesting bugs have been +found by modifying the target programs to call `abort()` when say: + +- Two bignum libraries produce different outputs when given the same + fuzzer-generated input. + +- An image library produces different outputs when asked to decode the same + input image several times in a row. + +- A serialization/deserialization library fails to produce stable outputs when + iteratively serializing and deserializing fuzzer-supplied data. + +- A compression library produces an output inconsistent with the input file when + asked to compress and then decompress a particular blob. + +Implementing these or similar sanity checks usually takes very little time; if +you are the maintainer of a particular package, you can make this code +conditional with `#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` (a flag also +shared with libfuzzer and honggfuzz) or `#ifdef __AFL_COMPILER` (this one is +just for AFL++). + +#### j) Known limitations & areas for improvement + +Here are some of the most important caveats for AFL++: + +- AFL++ detects faults by checking for the first spawned process dying due to a + signal (SIGSEGV, SIGABRT, etc). Programs that install custom handlers for + these signals may need to have the relevant code commented out. In the same + vein, faults in child processes spawned by the fuzzed target may evade + detection unless you manually add some code to catch that. + +- As with any other brute-force tool, the fuzzer offers limited coverage if + encryption, checksums, cryptographic signatures, or compression are used to + wholly wrap the actual data format to be tested. + + To work around this, you can comment out the relevant checks (see + utils/libpng_no_checksum/ for inspiration); if this is not possible, you can + also write a postprocessor, one of the hooks of custom mutators. See + [custom_mutators.md](custom_mutators.md) on how to use + `AFL_CUSTOM_MUTATOR_LIBRARY`. + +- There are some unfortunate trade-offs with ASAN and 64-bit binaries. This + isn't due to any specific fault of afl-fuzz. + +- There is no direct support for fuzzing network services, background daemons, + or interactive apps that require UI interaction to work. You may need to make + simple code changes to make them behave in a more traditional way. Preeny may + offer a relatively simple option, too - see: + [https://github.com/zardus/preeny](https://github.com/zardus/preeny) + + Some useful tips for modifying network-based services can be also found at: + [https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop](https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop) + +- Occasionally, sentient machines rise against their creators. If this happens + to you, please consult + [https://lcamtuf.coredump.cx/prep/](https://lcamtuf.coredump.cx/prep/). + +Beyond this, see [INSTALL.md](INSTALL.md) for platform-specific tips. + +### 4. Triaging crashes + +The coverage-based grouping of crashes usually produces a small data set that +can be quickly triaged manually or with a very simple GDB or Valgrind script. +Every crash is also traceable to its parent non-crashing test case in the queue, +making it easier to diagnose faults. + +Having said that, it's important to acknowledge that some fuzzing crashes can be +difficult to quickly evaluate for exploitability without a lot of debugging and +code analysis work. To assist with this task, afl-fuzz supports a very unique +"crash exploration" mode enabled with the -C flag. + +In this mode, the fuzzer takes one or more crashing test cases as the input and +uses its feedback-driven fuzzing strategies to very quickly enumerate all code +paths that can be reached in the program while keeping it in the crashing state. + +Mutations that do not result in a crash are rejected; so are any changes that do +not affect the execution path. + +The output is a small corpus of files that can be very rapidly examined to see +what degree of control the attacker has over the faulting address, or whether it +is possible to get past an initial out-of-bounds read - and see what lies +beneath. + +Oh, one more thing: for test case minimization, give afl-tmin a try. The tool +can be operated in a very simple way: + +```shell +./afl-tmin -i test_case -o minimized_result -- /path/to/program [...] +``` + +The tool works with crashing and non-crashing test cases alike. In the crash +mode, it will happily accept instrumented and non-instrumented binaries. In the +non-crashing mode, the minimizer relies on standard AFL++ instrumentation to +make the file simpler without altering the execution path. + +The minimizer accepts the -m, -t, -f and @@ syntax in a manner compatible with +afl-fuzz. + +Another tool in AFL++ is the afl-analyze tool. It takes an input file, attempts +to sequentially flip bytes, and observes the behavior of the tested program. It +then color-codes the input based on which sections appear to be critical, and +which are not; while not bulletproof, it can often offer quick insights into +complex file formats. More info about its operation can be found near the end of +[technical_details.md](technical_details.md). + ### The End Check out the [FAQ](FAQ.md) if it maybe answers your question (that you might diff --git a/docs/limitations.md b/docs/limitations.md deleted file mode 100644 index 8172a902..00000000 --- a/docs/limitations.md +++ /dev/null @@ -1,37 +0,0 @@ -# Known limitations & areas for improvement - -Here are some of the most important caveats for AFL++: - -- AFL++ detects faults by checking for the first spawned process dying due to a - signal (SIGSEGV, SIGABRT, etc). Programs that install custom handlers for - these signals may need to have the relevant code commented out. In the same - vein, faults in child processes spawned by the fuzzed target may evade - detection unless you manually add some code to catch that. - -- As with any other brute-force tool, the fuzzer offers limited coverage if - encryption, checksums, cryptographic signatures, or compression are used to - wholly wrap the actual data format to be tested. - -To work around this, you can comment out the relevant checks (see -utils/libpng_no_checksum/ for inspiration); if this is not possible, you can -also write a postprocessor, one of the hooks of custom mutators. See -[custom_mutators.md](custom_mutators.md) on how to use -`AFL_CUSTOM_MUTATOR_LIBRARY`. - -- There are some unfortunate trade-offs with ASAN and 64-bit binaries. This - isn't due to any specific fault of afl-fuzz. - -- There is no direct support for fuzzing network services, background daemons, - or interactive apps that require UI interaction to work. You may need to make - simple code changes to make them behave in a more traditional way. Preeny may - offer a relatively simple option, too - see: - [https://github.com/zardus/preeny](https://github.com/zardus/preeny) - -Some useful tips for modifying network-based services can be also found at: -[https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop](https://www.fastly.com/blog/how-to-fuzz-server-american-fuzzy-lop) - -- Occasionally, sentient machines rise against their creators. If this happens - to you, please consult - [https://lcamtuf.coredump.cx/prep/](https://lcamtuf.coredump.cx/prep/). - -Beyond this, see [INSTALL.md](INSTALL.md) for platform-specific tips. \ No newline at end of file diff --git a/docs/triaging_crashes.md b/docs/triaging_crashes.md deleted file mode 100644 index 21ccecaa..00000000 --- a/docs/triaging_crashes.md +++ /dev/null @@ -1,46 +0,0 @@ -# Triaging crashes - -The coverage-based grouping of crashes usually produces a small data set that -can be quickly triaged manually or with a very simple GDB or Valgrind script. -Every crash is also traceable to its parent non-crashing test case in the -queue, making it easier to diagnose faults. - -Having said that, it's important to acknowledge that some fuzzing crashes can be -difficult to quickly evaluate for exploitability without a lot of debugging and -code analysis work. To assist with this task, afl-fuzz supports a very unique -"crash exploration" mode enabled with the -C flag. - -In this mode, the fuzzer takes one or more crashing test cases as the input -and uses its feedback-driven fuzzing strategies to very quickly enumerate all -code paths that can be reached in the program while keeping it in the -crashing state. - -Mutations that do not result in a crash are rejected; so are any changes that -do not affect the execution path. - -The output is a small corpus of files that can be very rapidly examined to see -what degree of control the attacker has over the faulting address, or whether -it is possible to get past an initial out-of-bounds read - and see what lies -beneath. - -Oh, one more thing: for test case minimization, give afl-tmin a try. The tool -can be operated in a very simple way: - -```shell -./afl-tmin -i test_case -o minimized_result -- /path/to/program [...] -``` - -The tool works with crashing and non-crashing test cases alike. In the crash -mode, it will happily accept instrumented and non-instrumented binaries. In the -non-crashing mode, the minimizer relies on standard AFL++ instrumentation to make -the file simpler without altering the execution path. - -The minimizer accepts the -m, -t, -f and @@ syntax in a manner compatible with -afl-fuzz. - -Another tool in AFL++ is the afl-analyze tool. It takes an input -file, attempts to sequentially flip bytes, and observes the behavior of the -tested program. It then color-codes the input based on which sections appear to -be critical, and which are not; while not bulletproof, it can often offer quick -insights into complex file formats. More info about its operation can be found -near the end of [technical_details.md](technical_details.md). \ No newline at end of file -- cgit 1.4.1 From f11cf068dca784831d1c70e95258e85f5b1e64eb Mon Sep 17 00:00:00 2001 From: llzmb <46303940+llzmb@users.noreply.github.com> Date: Wed, 24 Nov 2021 10:52:29 +0100 Subject: Merge "common_sense_risks.md" into "fuzzing_in_depth.md" --- README.md | 4 ++-- docs/common_sense_risks.md | 36 ------------------------------------ docs/fuzzing_in_depth.md | 37 +++++++++++++++++++++++++++++++++++++ 3 files changed, 39 insertions(+), 38 deletions(-) delete mode 100644 docs/common_sense_risks.md (limited to 'docs/fuzzing_in_depth.md') diff --git a/README.md b/README.md index e0cb4558..989e8fdb 100644 --- a/README.md +++ b/README.md @@ -60,8 +60,8 @@ To build AFL++ yourself, continue at [docs/INSTALL.md](docs/INSTALL.md). ## Quick start: Fuzzing with AFL++ -*NOTE: Before you start, please read about the [common sense risks of -fuzzing](docs/common_sense_risks.md).* +*NOTE: Before you start, please read about the +[common sense risks of fuzzing](docs/fuzzing_in_depth.md#0-common-sense-risks).* This is a quick start for fuzzing targets with the source code available. To read about the process in detail, see diff --git a/docs/common_sense_risks.md b/docs/common_sense_risks.md deleted file mode 100644 index a8d68d7a..00000000 --- a/docs/common_sense_risks.md +++ /dev/null @@ -1,36 +0,0 @@ -# Common sense risks - -Please keep in mind that, similarly to many other computationally-intensive -tasks, fuzzing may put a strain on your hardware and on the OS. In particular: - - - Your CPU will run hot and will need adequate cooling. In most cases, if - cooling is insufficient or stops working properly, CPU speeds will be - automatically throttled. That said, especially when fuzzing on less - suitable hardware (laptops, smartphones, etc), it's not entirely impossible - for something to blow up. - - - Targeted programs may end up erratically grabbing gigabytes of memory or - filling up disk space with junk files. AFL++ tries to enforce basic memory - limits, but can't prevent each and every possible mishap. The bottom line - is that you shouldn't be fuzzing on systems where the prospect of data loss - is not an acceptable risk. - - - Fuzzing involves billions of reads and writes to the filesystem. On modern - systems, this will be usually heavily cached, resulting in fairly modest - "physical" I/O - but there are many factors that may alter this equation. - It is your responsibility to monitor for potential trouble; with very heavy - I/O, the lifespan of many HDDs and SSDs may be reduced. - - A good way to monitor disk I/O on Linux is the 'iostat' command: - -```shell - $ iostat -d 3 -x -k [...optional disk ID...] -``` - - Using the `AFL_TMPDIR` environment variable and a RAM-disk you can have the - heavy writing done in RAM to prevent the aforementioned wear and tear. For - example the following line will run a Docker container with all this preset: - - ```shell - # docker run -ti --mount type=tmpfs,destination=/ramdisk -e AFL_TMPDIR=/ramdisk aflplusplus/aflplusplus - ``` \ No newline at end of file diff --git a/docs/fuzzing_in_depth.md b/docs/fuzzing_in_depth.md index 4481bce6..19d8e783 100644 --- a/docs/fuzzing_in_depth.md +++ b/docs/fuzzing_in_depth.md @@ -13,6 +13,43 @@ Fuzzing source code is a three-step process: 3. Perform the fuzzing of the target by randomly mutating input and assessing if a generated input was processed in a new path in the target binary. +### 0. Common sense risks + +Please keep in mind that, similarly to many other computationally-intensive +tasks, fuzzing may put a strain on your hardware and on the OS. In particular: + +- Your CPU will run hot and will need adequate cooling. In most cases, if + cooling is insufficient or stops working properly, CPU speeds will be + automatically throttled. That said, especially when fuzzing on less suitable + hardware (laptops, smartphones, etc.), it's not entirely impossible for + something to blow up. + +- Targeted programs may end up erratically grabbing gigabytes of memory or + filling up disk space with junk files. AFL++ tries to enforce basic memory + limits, but can't prevent each and every possible mishap. The bottom line is + that you shouldn't be fuzzing on systems where the prospect of data loss is + not an acceptable risk. + +- Fuzzing involves billions of reads and writes to the filesystem. On modern + systems, this will be usually heavily cached, resulting in fairly modest + "physical" I/O - but there are many factors that may alter this equation. It + is your responsibility to monitor for potential trouble; with very heavy I/O, + the lifespan of many HDDs and SSDs may be reduced. + + A good way to monitor disk I/O on Linux is the `iostat` command: + + ```shell + $ iostat -d 3 -x -k [...optional disk ID...] + ``` + + Using the `AFL_TMPDIR` environment variable and a RAM-disk, you can have the + heavy writing done in RAM to prevent the aforementioned wear and tear. For + example, the following line will run a Docker container with all this preset: + + ```shell + # docker run -ti --mount type=tmpfs,destination=/ramdisk -e AFL_TMPDIR=/ramdisk aflplusplus/aflplusplus + ``` + ### 1. Instrumenting the target #### a) Selecting the best AFL++ compiler for instrumenting the target -- cgit 1.4.1 From 5b480f94511129e56062976d2c83daedd4a5043b Mon Sep 17 00:00:00 2001 From: llzmb <46303940+llzmb@users.noreply.github.com> Date: Wed, 24 Nov 2021 11:00:03 +0100 Subject: Edit "fuzzing_in_depth.md" --- docs/fuzzing_in_depth.md | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-) (limited to 'docs/fuzzing_in_depth.md') diff --git a/docs/fuzzing_in_depth.md b/docs/fuzzing_in_depth.md index 19d8e783..2365c6fd 100644 --- a/docs/fuzzing_in_depth.md +++ b/docs/fuzzing_in_depth.md @@ -336,22 +336,19 @@ target as possible improves the efficiency a lot. To operate correctly, the fuzzer requires one or more starting files that contain a good example of the input data normally expected by the targeted -application. There are two basic rules: +application. -- Keep the files small. Under 1 kB is ideal, although not strictly necessary. - For a discussion of why size matters, see [perf_tips.md](perf_tips.md). +Try to gather valid inputs for the target from wherever you can. E.g., if it is +the PNG picture format, try to find as many PNG files as possible, e.g., from +reported bugs, test suites, random downloads from the internet, unit test case +data - from all kind of PNG software. -- Use multiple test cases only if they are functionally different from each - other. There is no point in using fifty different vacation photos to fuzz an - image library. +If the input format is not known, you can also modify a target program to write +normal data it receives and processes to a file and use these. You can find many good examples of starting files in the [testcases/](../testcases) subdirectory that comes with this tool. -PS. If a large corpus of data is available for screening, you may want to use -the afl-cmin utility to identify a subset of functionally distinct files that -exercise different code paths in the target binary. - #### b) Making the input corpus unique Use the AFL++ tool `afl-cmin` to remove inputs from the corpus that do not @@ -787,8 +784,7 @@ Another tool in AFL++ is the afl-analyze tool. It takes an input file, attempts to sequentially flip bytes, and observes the behavior of the tested program. It then color-codes the input based on which sections appear to be critical, and which are not; while not bulletproof, it can often offer quick insights into -complex file formats. More info about its operation can be found near the end of -[technical_details.md](technical_details.md). +complex file formats. ### The End -- cgit 1.4.1 From fce93647cc788683be3d8cca79c4689de4b71c3f Mon Sep 17 00:00:00 2001 From: llzmb <46303940+llzmb@users.noreply.github.com> Date: Wed, 24 Nov 2021 13:24:12 +0100 Subject: Merge "perf_tips.md" into "best_practices.md" and "fuzzing_in_depth.md" --- docs/best_practices.md | 4 +- docs/fuzzing_in_depth.md | 46 +++++++---- docs/perf_tips.md | 209 ----------------------------------------------- 3 files changed, 32 insertions(+), 227 deletions(-) delete mode 100644 docs/perf_tips.md (limited to 'docs/fuzzing_in_depth.md') diff --git a/docs/best_practices.md b/docs/best_practices.md index 5f2d45ed..979849f4 100644 --- a/docs/best_practices.md +++ b/docs/best_practices.md @@ -64,11 +64,11 @@ which allows you to define network state with different type of data packets. 1. Use [llvm_mode](../instrumentation/README.llvm.md): afl-clang-lto (llvm >= 11) or afl-clang-fast (llvm >= 9 recommended). 2. Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 speed increase). -3. Use the [AFL++ snapshot module](https://github.com/AFLplusplus/AFL-Snapshot-LKM) (x2 speed increase). +3. Instrument just what you are interested in, see [instrumentation/README.instrument_list.md](../instrumentation/README.instrument_list.md). 4. If you do not use shmem persistent mode, use `AFL_TMPDIR` to put the input file directory on a tempfs location, see [env_variables.md](env_variables.md). 5. Improve Linux kernel performance: modify `/etc/default/grub`, set `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then `update-grub` and `reboot` (warning: makes the system less secure). 6. Running on an `ext2` filesystem with `noatime` mount option will be a bit faster than on any other journaling filesystem. -7. Use your cores ([fuzzing_in_depth.md:b) Using multiple cores](fuzzing_in_depth.md#b-using-multiple-cores))! +7. Use your cores ([fuzzing_in_depth.md:3c) Using multiple cores](fuzzing_in_depth.md#c-using-multiple-cores))! ### Improving stability diff --git a/docs/fuzzing_in_depth.md b/docs/fuzzing_in_depth.md index 2365c6fd..869ed212 100644 --- a/docs/fuzzing_in_depth.md +++ b/docs/fuzzing_in_depth.md @@ -419,7 +419,7 @@ as test data in there. If you do not want anything special, the defaults are already usually best, hence all you need is to specify the seed input directory with the result of -step [2a. Collect inputs](#a-collect-inputs): +step [2a) Collect inputs](#a-collect-inputs): `afl-fuzz -i input -o output -- bin/target -d @@` Note that the directory specified with -o will be created if it does not exist. @@ -438,11 +438,6 @@ If you need to stop and re-start the fuzzing, use the same command line options mode!) and switch the input directory with a dash (`-`): `afl-fuzz -i - -o output -- bin/target -d @@` -Memory limits are not enforced by afl-fuzz by default and the system may run out -of memory. You can decrease the memory with the `-m` option, the value is in MB. -If this is too small for the target, you can usually see this by afl-fuzz -bailing with the message that it could not connect to the forkserver. - Adding a dictionary is helpful. See the directory [dictionaries/](../dictionaries/) if something is already included for your data format, and tell afl-fuzz to load that dictionary by adding `-x @@ -472,7 +467,26 @@ is: All labels are explained in [status_screen.md](status_screen.md). -#### b) Using multiple cores +#### b) Keeping memory use and timeouts in check + +Memory limits are not enforced by afl-fuzz by default and the system may run out +of memory. You can decrease the memory with the `-m` option, the value is in MB. +If this is too small for the target, you can usually see this by afl-fuzz +bailing with the message that it could not connect to the forkserver. + +Consider setting low values for `-m` and `-t`. + +For programs that are nominally very fast, but get sluggish for some inputs, you +can also try setting `-t` values that are more punishing than what `afl-fuzz` +dares to use on its own. On fast and idle machines, going down to `-t 5` may be +a viable plan. + +The `-m` parameter is worth looking at, too. Some programs can end up spending a +fair amount of time allocating and initializing megabytes of memory when +presented with pathological inputs. Low `-m` values can make them give up sooner +and not waste CPU time. + +#### c) Using multiple cores If you want to seriously fuzz then use as many cores/threads as possible to fuzz your target. @@ -537,7 +551,7 @@ directory of a different fuzzer is, e.g. `-F /src/target/honggfuzz`. Using honggfuzz (with `-n 1` or `-n 2`) and libfuzzer in parallel is highly recommended! -#### c) Using multiple machines for fuzzing +#### d) Using multiple machines for fuzzing Maybe you have more than one machine you want to fuzz the same target on. Simply start the `afl-fuzz` (and perhaps libfuzzer, honggfuzz, ...) @@ -575,7 +589,7 @@ done You can run this manually, per cron job - as you need it. There is a more complex and configurable script in `utils/distributed_fuzzing`. -#### d) The status of the fuzz campaign +#### e) The status of the fuzz campaign AFL++ comes with the `afl-whatsup` script to show the status of the fuzzing campaign. @@ -593,7 +607,7 @@ afl-plot, which generates an index.html file and a graphs that show how the fuzzing instance is performing. The syntax is `afl-plot instance_dir web_dir`, e.g., `afl-plot out/default /srv/www/htdocs/plot`. -#### e) Stopping fuzzing, restarting fuzzing, adding new seeds +#### f) Stopping fuzzing, restarting fuzzing, adding new seeds To stop an afl-fuzz run, simply press Control-C. @@ -608,7 +622,7 @@ are in `newseeds/` directory: AFL_BENCH_JUST_ONE=1 AFL_FAST_CAL=1 afl-fuzz -i newseeds -o out -S newseeds -- ./target ``` -#### f) Checking the coverage of the fuzzing +#### g) Checking the coverage of the fuzzing The `paths found` value is a bad indicator for checking how good the coverage is. @@ -648,7 +662,7 @@ individual fuzzing campaigns each with one of these options set. E.g., if you fuzz a library to convert image formats and your target is the png to tiff API then you will not touch any of the other library APIs and features. -#### g) How long to fuzz a target? +#### h) How long to fuzz a target? This is a difficult question. Basically if no new path is found for a long time (e.g. for a day or a week) then you can expect that your fuzzing won't be @@ -660,7 +674,7 @@ Keep the queue/ directory (for future fuzzings of the same or similar targets) and use them to seed other good fuzzers like libfuzzer with the -entropic switch or honggfuzz. -#### h) Improve the speed! +#### i) Improve the speed! * Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 speed increase) @@ -675,11 +689,11 @@ or honggfuzz. also just run `sudo afl-persistent-config` * Linux: Running on an `ext2` filesystem with `noatime` mount option will be a bit faster than on any other journaling filesystem -* Use your cores! [b) Using multiple cores](#b-using-multiple-cores) +* Use your cores! [3c) Using multiple cores](#c-using-multiple-cores) * Run `sudo afl-system-config` before starting the first afl-fuzz instance after a reboot -#### i) Going beyond crashes +#### j) Going beyond crashes Fuzzing is a wonderful and underutilized technique for discovering non-crashing design and implementation errors, too. Quite a few interesting bugs have been @@ -703,7 +717,7 @@ conditional with `#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` (a flag also shared with libfuzzer and honggfuzz) or `#ifdef __AFL_COMPILER` (this one is just for AFL++). -#### j) Known limitations & areas for improvement +#### k) Known limitations & areas for improvement Here are some of the most important caveats for AFL++: diff --git a/docs/perf_tips.md b/docs/perf_tips.md deleted file mode 100644 index 1e8fd4d0..00000000 --- a/docs/perf_tips.md +++ /dev/null @@ -1,209 +0,0 @@ -## Tips for performance optimization - - This file provides tips for troubleshooting slow or wasteful fuzzing jobs. - See README.md for the general instruction manual. - -## 1. Keep your test cases small - -This is probably the single most important step to take! Large test cases do -not merely take more time and memory to be parsed by the tested binary, but -also make the fuzzing process dramatically less efficient in several other -ways. - -To illustrate, let's say that you're randomly flipping bits in a file, one bit -at a time. Let's assume that if you flip bit #47, you will hit a security bug; -flipping any other bit just results in an invalid document. - -Now, if your starting test case is 100 bytes long, you will have a 71% chance of -triggering the bug within the first 1,000 execs - not bad! But if the test case -is 1 kB long, the probability that we will randomly hit the right pattern in -the same timeframe goes down to 11%. And if it has 10 kB of non-essential -cruft, the odds plunge to 1%. - -On top of that, with larger inputs, the binary may be now running 5-10x times -slower than before - so the overall drop in fuzzing efficiency may be easily -as high as 500x or so. - -In practice, this means that you shouldn't fuzz image parsers with your -vacation photos. Generate a tiny 16x16 picture instead, and run it through -`jpegtran` or `pngcrunch` for good measure. The same goes for most other types -of documents. - -There's plenty of small starting test cases in ../testcases/ - try them out -or submit new ones! - -If you want to start with a larger, third-party corpus, run `afl-cmin` with an -aggressive timeout on that data set first. - -## 2. Use a simpler target - -Consider using a simpler target binary in your fuzzing work. For example, for -image formats, bundled utilities such as `djpeg`, `readpng`, or `gifhisto` are -considerably (10-20x) faster than the convert tool from ImageMagick - all while exercising roughly the same library-level image parsing code. - -Even if you don't have a lightweight harness for a particular target, remember -that you can always use another, related library to generate a corpus that will -be then manually fed to a more resource-hungry program later on. - -Also note that reading the fuzzing input via stdin is faster than reading from -a file. - -## 3. Use LLVM persistent instrumentation - -The LLVM mode offers a "persistent", in-process fuzzing mode that can -work well for certain types of self-contained libraries, and for fast targets, -can offer performance gains up to 5-10x; and a "deferred fork server" mode -that can offer huge benefits for programs with high startup overhead. Both -modes require you to edit the source code of the fuzzed program, but the -changes often amount to just strategically placing a single line or two. - -If there are important data comparisons performed (e.g. `strcmp(ptr, MAGIC_HDR)`) -then using laf-intel (see instrumentation/README.laf-intel.md) will help `afl-fuzz` a lot -to get to the important parts in the code. - -If you are only interested in specific parts of the code being fuzzed, you can -instrument_files the files that are actually relevant. This improves the speed and -accuracy of afl. See instrumentation/README.instrument_list.md - -## 4. Profile and optimize the binary - -Check for any parameters or settings that obviously improve performance. For -example, the djpeg utility that comes with IJG jpeg and libjpeg-turbo can be -called with: - -```bash - -dct fast -nosmooth -onepass -dither none -scale 1/4 -``` - -...and that will speed things up. There is a corresponding drop in the quality -of decoded images, but it's probably not something you care about. - -In some programs, it is possible to disable output altogether, or at least use -an output format that is computationally inexpensive. For example, with image -transcoding tools, converting to a BMP file will be a lot faster than to PNG. - -With some laid-back parsers, enabling "strict" mode (i.e., bailing out after -first error) may result in smaller files and improved run time without -sacrificing coverage; for example, for sqlite, you may want to specify -bail. - -If the program is still too slow, you can use `strace -tt` or an equivalent -profiling tool to see if the targeted binary is doing anything silly. -Sometimes, you can speed things up simply by specifying `/dev/null` as the -config file, or disabling some compile-time features that aren't really needed -for the job (try `./configure --help`). One of the notoriously resource-consuming -things would be calling other utilities via `exec*()`, `popen()`, `system()`, or -equivalent calls; for example, tar can invoke external decompression tools -when it decides that the input file is a compressed archive. - -Some programs may also intentionally call `sleep()`, `usleep()`, or `nanosleep()`; -vim is a good example of that. Other programs may attempt `fsync()` and so on. -There are third-party libraries that make it easy to get rid of such code, -e.g.: - - https://launchpad.net/libeatmydata - -In programs that are slow due to unavoidable initialization overhead, you may -want to try the LLVM deferred forkserver mode (see README.llvm.md), -which can give you speed gains up to 10x, as mentioned above. - -Last but not least, if you are using ASAN and the performance is unacceptable, -consider turning it off for now, and manually examining the generated corpus -with an ASAN-enabled binary later on. - -## 5. Instrument just what you need - -Instrument just the libraries you actually want to stress-test right now, one -at a time. Let the program use system-wide, non-instrumented libraries for -any functionality you don't actually want to fuzz. For example, in most -cases, it doesn't make to instrument `libgmp` just because you're testing a -crypto app that relies on it for bignum math. - -Beware of programs that come with oddball third-party libraries bundled with -their source code (Spidermonkey is a good example of this). Check `./configure` -options to use non-instrumented system-wide copies instead. - -## 6. Parallelize your fuzzers - -The fuzzer is designed to need ~1 core per job. This means that on a, say, -4-core system, you can easily run four parallel fuzzing jobs with relatively -little performance hit. For tips on how to do that, see parallel_fuzzing.md. - -The `afl-gotcpu` utility can help you understand if you still have idle CPU -capacity on your system. (It won't tell you about memory bandwidth, cache -misses, or similar factors, but they are less likely to be a concern.) - -## 7. Keep memory use and timeouts in check - -Consider setting low values for `-m` and `-t`. - -For programs that are nominally very fast, but get sluggish for some inputs, -you can also try setting `-t` values that are more punishing than what `afl-fuzz` -dares to use on its own. On fast and idle machines, going down to `-t 5` may be -a viable plan. - -The `-m` parameter is worth looking at, too. Some programs can end up spending -a fair amount of time allocating and initializing megabytes of memory when -presented with pathological inputs. Low `-m` values can make them give up sooner -and not waste CPU time. - -## 8. Check OS configuration - -There are several OS-level factors that may affect fuzzing speed: - - - If you have no risk of power loss then run your fuzzing on a tmpfs - partition. This increases the performance noticably. - Alternatively you can use `AFL_TMPDIR` to point to a tmpfs location to - just write the input file to a tmpfs. - - High system load. Use idle machines where possible. Kill any non-essential - CPU hogs (idle browser windows, media players, complex screensavers, etc). - - Network filesystems, either used for fuzzer input / output, or accessed by - the fuzzed binary to read configuration files (pay special attention to the - home directory - many programs search it for dot-files). - - Disable all the spectre, meltdown etc. security countermeasures in the - kernel if your machine is properly separated: - -``` -ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off -no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable -nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off -spectre_v2=off stf_barrier=off -``` - In most Linux distributions you can put this into a `/etc/default/grub` - variable. - You can use `sudo afl-persistent-config` to set these options for you. - -The following list of changes are made when executing `afl-system-config`: - - - On-demand CPU scaling. The Linux `ondemand` governor performs its analysis - on a particular schedule and is known to underestimate the needs of - short-lived processes spawned by `afl-fuzz` (or any other fuzzer). On Linux, - this can be fixed with: - -``` bash - cd /sys/devices/system/cpu - echo performance | tee cpu*/cpufreq/scaling_governor -``` - - On other systems, the impact of CPU scaling will be different; when fuzzing, - use OS-specific tools to find out if all cores are running at full speed. - - Transparent huge pages. Some allocators, such as `jemalloc`, can incur a - heavy fuzzing penalty when transparent huge pages (THP) are enabled in the - kernel. You can disable this via: - -```bash - echo never > /sys/kernel/mm/transparent_hugepage/enabled -``` - - - Suboptimal scheduling strategies. The significance of this will vary from - one target to another, but on Linux, you may want to make sure that the - following options are set: - -```bash - echo 1 >/proc/sys/kernel/sched_child_runs_first - echo 1 >/proc/sys/kernel/sched_autogroup_enabled -``` - - Setting a different scheduling policy for the fuzzer process - say - `SCHED_RR` - can usually speed things up, too, but needs to be done with - care. - -- cgit 1.4.1 From b8a883787501180e8ead0e5f21e8e858841be73b Mon Sep 17 00:00:00 2001 From: llzmb <46303940+llzmb@users.noreply.github.com> Date: Thu, 25 Nov 2021 21:00:39 +0100 Subject: Fix typo (#1183) --- docs/fuzzing_in_depth.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs/fuzzing_in_depth.md') diff --git a/docs/fuzzing_in_depth.md b/docs/fuzzing_in_depth.md index 869ed212..2a423db7 100644 --- a/docs/fuzzing_in_depth.md +++ b/docs/fuzzing_in_depth.md @@ -516,7 +516,7 @@ For every secondary fuzzer there should be a variation, e.g.: * one to three fuzzers should fuzz a target compiled with laf-intel/COMPCOV (see above). Important note: If you run more than one laf-intel/COMPCOV fuzzer and you want them to share their intermediate results, the main fuzzer (`-M`) must - be one of the them! (Although this is not really recommended.) + be one of them! (Although this is not really recommended.) All other secondaries should be used like this: * a quarter to a third with the MOpt mutator enabled: `-L 0` -- cgit 1.4.1 From 2412ff63e3f86f0d7876a550e64f2482e85a77c6 Mon Sep 17 00:00:00 2001 From: llzmb <46303940+llzmb@users.noreply.github.com> Date: Fri, 26 Nov 2021 12:50:40 +0100 Subject: Merge "ci_fuzzing.md" into "fuzzing_in_depth.md" --- docs/ci_fuzzing.md | 29 ------------- docs/fuzzing_in_depth.md | 103 ++++++++++++++++++++++++++++++++--------------- 2 files changed, 71 insertions(+), 61 deletions(-) delete mode 100644 docs/ci_fuzzing.md (limited to 'docs/fuzzing_in_depth.md') diff --git a/docs/ci_fuzzing.md b/docs/ci_fuzzing.md deleted file mode 100644 index 8d1a2f99..00000000 --- a/docs/ci_fuzzing.md +++ /dev/null @@ -1,29 +0,0 @@ -# CI Fuzzing - -Some notes on CI Fuzzing - this fuzzing is different to normal fuzzing campaigns as these are much shorter runnings. - -1. Always: - * LTO has a much longer compile time which is diametrical to short fuzzing - hence use afl-clang-fast instead. - * If you compile with CMPLOG then you can save fuzzing time and reuse that compiled target for both the -c option and the main fuzz target. - This will impact the speed by ~15% though. - * `AFL_FAST_CAL` - Enable fast calibration, this halfs the time the saturated corpus needs to be loaded. - * `AFL_CMPLOG_ONLY_NEW` - only perform cmplog on new found paths, not the initial corpus as this very likely has been done for them already. - * Keep the generated corpus, use afl-cmin and reuse it every time! - -2. Additionally randomize the AFL++ compilation options, e.g. - * 40% for `AFL_LLVM_CMPLOG` - * 10% for `AFL_LLVM_LAF_ALL` - -3. Also randomize the afl-fuzz runtime options, e.g. - * 65% for `AFL_DISABLE_TRIM` - * 50% use a dictionary generated by `AFL_LLVM_DICT2FILE` - * 40% use MOpt (`-L 0`) - * 40% for `AFL_EXPAND_HAVOC_NOW` - * 20% for old queue processing (`-Z`) - * for CMPLOG targets, 60% for `-l 2`, 40% for `-l 3` - -4. Do *not* run any `-M` modes, just running `-S` modes is better for CI fuzzing. -`-M` enables old queue handling etc. which is good for a fuzzing campaign but not good for short CI runs. - -How this can look like can e.g. be seen at AFL++'s setup in Google's [oss-fuzz](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/compile_afl) -and [clusterfuzz](https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/bot/fuzzers/afl/launcher.py). diff --git a/docs/fuzzing_in_depth.md b/docs/fuzzing_in_depth.md index 2a423db7..251bbc1d 100644 --- a/docs/fuzzing_in_depth.md +++ b/docs/fuzzing_in_depth.md @@ -13,7 +13,7 @@ Fuzzing source code is a three-step process: 3. Perform the fuzzing of the target by randomly mutating input and assessing if a generated input was processed in a new path in the target binary. -### 0. Common sense risks +## 0. Common sense risks Please keep in mind that, similarly to many other computationally-intensive tasks, fuzzing may put a strain on your hardware and on the OS. In particular: @@ -50,9 +50,9 @@ tasks, fuzzing may put a strain on your hardware and on the OS. In particular: # docker run -ti --mount type=tmpfs,destination=/ramdisk -e AFL_TMPDIR=/ramdisk aflplusplus/aflplusplus ``` -### 1. Instrumenting the target +## 1. Instrumenting the target -#### a) Selecting the best AFL++ compiler for instrumenting the target +### a) Selecting the best AFL++ compiler for instrumenting the target AFL++ comes with a central compiler `afl-cc` that incorporates various different kinds of compiler targets and and instrumentation options. The following @@ -111,7 +111,7 @@ command), the compile-time tools make fairly broad use of environment variables, which can be listed with `afl-cc -hh` or by reading [env_variables.md](env_variables.md). -#### b) Selecting instrumentation options +### b) Selecting instrumentation options The following options are available when you instrument with LTO mode (afl-clang-fast/afl-clang-lto): @@ -160,7 +160,7 @@ AFL++ performs "never zero" counting in its bitmap. You can read more about this here: * [instrumentation/README.neverzero.md](../instrumentation/README.neverzero.md) -#### c) Selecting sanitizers +### c) Selecting sanitizers It is possible to use sanitizers when instrumenting targets for fuzzing, which allows you to find bugs that would not necessarily result in a crash. @@ -208,7 +208,7 @@ CFISAN. You might need to experiment which sanitizers you can combine in a target (which means more instances can be run without a sanitized target, which is more effective). -#### d) Modifying the target +### d) Modifying the target If the target has features that make fuzzing more difficult, e.g. checksums, HMAC, etc. then modify the source code so that checks for these values are @@ -225,7 +225,7 @@ products by eliminating these checks within these AFL specific blocks: All AFL++ compilers will set this preprocessor definition automatically. -#### e) Instrumenting the target +### e) Instrumenting the target In this step the target source code is compiled so that it can be fuzzed. @@ -256,7 +256,7 @@ Then build the target. (Usually with `make`) aborts then set `export AFL_NOOPT=1` which will then just behave like the real compiler. This option has to be unset again before building the target! -##### configure +#### configure For `configure` build systems this is usually done by: `CC=afl-clang-fast CXX=afl-clang-fast++ ./configure --disable-shared` @@ -265,7 +265,7 @@ Note that if you are using the (better) afl-clang-lto compiler you also have to set AR to llvm-ar[-VERSION] and RANLIB to llvm-ranlib[-VERSION] - as is described in [instrumentation/README.lto.md](../instrumentation/README.lto.md). -##### cmake +#### cmake For `cmake` build systems this is usually done by: `mkdir build; cd build; cmake -DCMAKE_C_COMPILER=afl-cc -DCMAKE_CXX_COMPILER=afl-c++ ..` @@ -274,12 +274,12 @@ Note that if you are using the (better) afl-clang-lto compiler you also have to set AR to llvm-ar[-VERSION] and RANLIB to llvm-ranlib[-VERSION] - as is described in [instrumentation/README.lto.md](../instrumentation/README.lto.md). -##### meson +#### meson For meson you have to set the AFL++ compiler with the very first command! `CC=afl-cc CXX=afl-c++ meson` -##### other build systems or if configure/cmake didn't work +#### other build systems or if configure/cmake didn't work Sometimes cmake and configure do not pick up the AFL++ compiler, or the ranlib/ar that is needed - because this was just not foreseen by the developer @@ -288,7 +288,7 @@ non-standard way to set this, otherwise set up the build normally and edit the generated build environment afterwards manually to point it to the right compiler (and/or ranlib and ar). -#### f) Better instrumentation +### f) Better instrumentation If you just fuzz a target program as-is you are wasting a great opportunity for much more fuzzing speed. @@ -305,7 +305,7 @@ for details. Basically if you do not fuzz a target in persistent mode then you are just doing it for a hobby and not professionally :-). -#### g) libfuzzer fuzzer harnesses with LLVMFuzzerTestOneInput() +### g) libfuzzer fuzzer harnesses with LLVMFuzzerTestOneInput() libfuzzer `LLVMFuzzerTestOneInput()` harnesses are the defacto standard for fuzzing, and they can be used with AFL++ (and honggfuzz) as well! @@ -327,12 +327,12 @@ shared-memory test cases and hence gives you the fastest speed possible. For more information, see [utils/aflpp_driver/README.md](../utils/aflpp_driver/README.md). -### 2. Preparing the fuzzing campaign +## 2. Preparing the fuzzing campaign As you fuzz the target with mutated input, having as diverse inputs for the target as possible improves the efficiency a lot. -#### a) Collecting inputs +### a) Collecting inputs To operate correctly, the fuzzer requires one or more starting files that contain a good example of the input data normally expected by the targeted @@ -349,7 +349,7 @@ normal data it receives and processes to a file and use these. You can find many good examples of starting files in the [testcases/](../testcases) subdirectory that comes with this tool. -#### b) Making the input corpus unique +### b) Making the input corpus unique Use the AFL++ tool `afl-cmin` to remove inputs from the corpus that do not produce a new path in the target. @@ -366,7 +366,7 @@ default. This step is highly recommended! -#### c) Minimizing all corpus files +### c) Minimizing all corpus files The shorter the input files that still traverse the same path within the target, the better the fuzzing will be. This minimization is done with `afl-tmin` @@ -383,13 +383,13 @@ done This step can also be parallelized, e.g. with `parallel`. Note that this step is rather optional though. -#### Done! +### Done! The INPUTS_UNIQUE/ directory from step b) - or even better the directory input/ if you minimized the corpus in step c) - is the resulting input corpus directory to be used in fuzzing! :-) -### 3. Fuzzing the target +## 3. Fuzzing the target In this final step we fuzz the target. There are not that many important options to run the target - unless you want to use many CPU cores/threads for the @@ -398,7 +398,7 @@ fuzzing, which will make the fuzzing much more useful. If you just use one CPU for fuzzing, then you are fuzzing just for fun and not seriously :-) -#### a) Running afl-fuzz +### a) Running afl-fuzz Before you do even a test run of afl-fuzz execute `sudo afl-system-config` (on the host if you execute afl-fuzz in a docker container). This reconfigures the @@ -467,7 +467,7 @@ is: All labels are explained in [status_screen.md](status_screen.md). -#### b) Keeping memory use and timeouts in check +### b) Keeping memory use and timeouts in check Memory limits are not enforced by afl-fuzz by default and the system may run out of memory. You can decrease the memory with the `-m` option, the value is in MB. @@ -486,7 +486,7 @@ fair amount of time allocating and initializing megabytes of memory when presented with pathological inputs. Low `-m` values can make them give up sooner and not waste CPU time. -#### c) Using multiple cores +### c) Using multiple cores If you want to seriously fuzz then use as many cores/threads as possible to fuzz your target. @@ -551,7 +551,7 @@ directory of a different fuzzer is, e.g. `-F /src/target/honggfuzz`. Using honggfuzz (with `-n 1` or `-n 2`) and libfuzzer in parallel is highly recommended! -#### d) Using multiple machines for fuzzing +### d) Using multiple machines for fuzzing Maybe you have more than one machine you want to fuzz the same target on. Simply start the `afl-fuzz` (and perhaps libfuzzer, honggfuzz, ...) @@ -589,7 +589,7 @@ done You can run this manually, per cron job - as you need it. There is a more complex and configurable script in `utils/distributed_fuzzing`. -#### e) The status of the fuzz campaign +### e) The status of the fuzz campaign AFL++ comes with the `afl-whatsup` script to show the status of the fuzzing campaign. @@ -607,7 +607,7 @@ afl-plot, which generates an index.html file and a graphs that show how the fuzzing instance is performing. The syntax is `afl-plot instance_dir web_dir`, e.g., `afl-plot out/default /srv/www/htdocs/plot`. -#### f) Stopping fuzzing, restarting fuzzing, adding new seeds +### f) Stopping fuzzing, restarting fuzzing, adding new seeds To stop an afl-fuzz run, simply press Control-C. @@ -622,7 +622,7 @@ are in `newseeds/` directory: AFL_BENCH_JUST_ONE=1 AFL_FAST_CAL=1 afl-fuzz -i newseeds -o out -S newseeds -- ./target ``` -#### g) Checking the coverage of the fuzzing +### g) Checking the coverage of the fuzzing The `paths found` value is a bad indicator for checking how good the coverage is. @@ -662,7 +662,7 @@ individual fuzzing campaigns each with one of these options set. E.g., if you fuzz a library to convert image formats and your target is the png to tiff API then you will not touch any of the other library APIs and features. -#### h) How long to fuzz a target? +### h) How long to fuzz a target? This is a difficult question. Basically if no new path is found for a long time (e.g. for a day or a week) then you can expect that your fuzzing won't be @@ -674,7 +674,7 @@ Keep the queue/ directory (for future fuzzings of the same or similar targets) and use them to seed other good fuzzers like libfuzzer with the -entropic switch or honggfuzz. -#### i) Improve the speed! +### i) Improve the speed! * Use [persistent mode](../instrumentation/README.persistent_mode.md) (x2-x20 speed increase) @@ -693,7 +693,7 @@ or honggfuzz. * Run `sudo afl-system-config` before starting the first afl-fuzz instance after a reboot -#### j) Going beyond crashes +### j) Going beyond crashes Fuzzing is a wonderful and underutilized technique for discovering non-crashing design and implementation errors, too. Quite a few interesting bugs have been @@ -717,7 +717,7 @@ conditional with `#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` (a flag also shared with libfuzzer and honggfuzz) or `#ifdef __AFL_COMPILER` (this one is just for AFL++). -#### k) Known limitations & areas for improvement +### k) Known limitations & areas for improvement Here are some of the most important caveats for AFL++: @@ -755,7 +755,7 @@ Here are some of the most important caveats for AFL++: Beyond this, see [INSTALL.md](INSTALL.md) for platform-specific tips. -### 4. Triaging crashes +## 4. Triaging crashes The coverage-based grouping of crashes usually produces a small data set that can be quickly triaged manually or with a very simple GDB or Valgrind script. @@ -800,7 +800,46 @@ then color-codes the input based on which sections appear to be critical, and which are not; while not bulletproof, it can often offer quick insights into complex file formats. -### The End + +## 5. CI fuzzing + +Some notes on CI fuzzing - this fuzzing is different to normal fuzzing campaigns +as these are much shorter runnings. + +1. Always: + * LTO has a much longer compile time which is diametrical to short fuzzing - + hence use afl-clang-fast instead. + * If you compile with CMPLOG, then you can save fuzzing time and reuse that + compiled target for both the `-c` option and the main fuzz target. This + will impact the speed by ~15% though. + * `AFL_FAST_CAL` - Enable fast calibration, this halves the time the + saturated corpus needs to be loaded. + * `AFL_CMPLOG_ONLY_NEW` - only perform cmplog on new found paths, not the + initial corpus as this very likely has been done for them already. + * Keep the generated corpus, use afl-cmin and reuse it every time! + +2. Additionally randomize the AFL++ compilation options, e.g.: + * 40% for `AFL_LLVM_CMPLOG` + * 10% for `AFL_LLVM_LAF_ALL` + +3. Also randomize the afl-fuzz runtime options, e.g.: + * 65% for `AFL_DISABLE_TRIM` + * 50% use a dictionary generated by `AFL_LLVM_DICT2FILE` + * 40% use MOpt (`-L 0`) + * 40% for `AFL_EXPAND_HAVOC_NOW` + * 20% for old queue processing (`-Z`) + * for CMPLOG targets, 60% for `-l 2`, 40% for `-l 3` + +4. Do *not* run any `-M` modes, just running `-S` modes is better for CI + fuzzing. `-M` enables old queue handling etc. which is good for a fuzzing + campaign but not good for short CI runs. + +How this can look like can, e.g., be seen at AFL++'s setup in Google's +[oss-fuzz](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/compile_afl) +and +[clusterfuzz](https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/bot/fuzzers/afl/launcher.py). + +## The End Check out the [FAQ](FAQ.md) if it maybe answers your question (that you might not even have known you had ;-) ). -- cgit 1.4.1