From b1822f26022c323e4b47198abbb7d5f5ea6a57e5 Mon Sep 17 00:00:00 2001 From: hexcoder- Date: Fri, 18 Oct 2019 18:01:33 +0200 Subject: reintegrate gcc_plugin --- gcc_plugin/README.gcc.md | 162 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 gcc_plugin/README.gcc.md (limited to 'gcc_plugin/README.gcc.md') diff --git a/gcc_plugin/README.gcc.md b/gcc_plugin/README.gcc.md new file mode 100644 index 00000000..2e19e911 --- /dev/null +++ b/gcc_plugin/README.gcc.md @@ -0,0 +1,162 @@ +=========================================== +GCC-based instrumentation for afl-fuzz +====================================== + + (See ../docs/README.md for the general instruction manual.) + (See ../llvm_mode/README.md for the LLVM-based instrumentation.) + +!!! TODO items are: +!!! => inline instrumentation has to work! +!!! + + +## 1) Introduction + +The code in this directory allows you to instrument programs for AFL using +true compiler-level instrumentation, instead of the more crude +assembly-level rewriting approach taken by afl-gcc and afl-clang. This has +several interesting properties: + + - The compiler can make many optimizations that are hard to pull off when + manually inserting assembly. As a result, some slow, CPU-bound programs will + run up to around faster. + + The gains are less pronounced for fast binaries, where the speed is limited + chiefly by the cost of creating new processes. In such cases, the gain will + probably stay within 10%. + + - The instrumentation is CPU-independent. At least in principle, you should + be able to rely on it to fuzz programs on non-x86 architectures (after + building afl-fuzz with AFL_NOX86=1). + + - Because the feature relies on the internals of GCC, it is gcc-specific + and will *not* work with LLVM (see ../llvm_mode for an alternative). + +Once this implementation is shown to be sufficiently robust and portable, it +will probably replace afl-gcc. For now, it can be built separately and +co-exists with the original code. + +The idea and much of the implementation comes from Laszlo Szekeres. + +## 2) How to use + +In order to leverage this mechanism, you need to have modern enough GCC +(>= version 4.5.0) and the plugin headers installed on your system. That +should be all you need. On Debian machines, these headers can be acquired by +installing the `gcc--plugin-dev` packages. + +To build the instrumentation itself, type 'make'. This will generate binaries +called afl-gcc-fast and afl-g++-fast in the parent directory. Once this +is done, you can instrument third-party code in a way similar to the standard +operating mode of AFL, e.g.: + + CC=/path/to/afl/afl-gcc-fast ./configure [...options...] + make + +Be sure to also include CXX set to afl-g++-fast for C++ code. + +The tool honors roughly the same environmental variables as afl-gcc (see +../docs/env_variables.txt). This includes AFL_INST_RATIO, AFL_USE_ASAN, +AFL_HARDEN, and AFL_DONT_OPTIMIZE. + +Note: if you want the GCC plugin to be installed on your system for all +users, you need to build it before issuing 'make install' in the parent +directory. + +## 3) Gotchas, feedback, bugs + +This is an early-stage mechanism, so field reports are welcome. You can send bug +reports to . + +## 4) Bonus feature #1: deferred initialization + +AFL tries to optimize performance by executing the targeted binary just once, +stopping it just before main(), and then cloning this "master" process to get +a steady supply of targets to fuzz. + +Although this approach eliminates much of the OS-, linker- and libc-level +costs of executing the program, it does not always help with binaries that +perform other time-consuming initialization steps - say, parsing a large config +file before getting to the fuzzed data. + +In such cases, it's beneficial to initialize the forkserver a bit later, once +most of the initialization work is already done, but before the binary attempts +to read the fuzzed input and parse it; in some cases, this can offer a 10x+ +performance gain. You can implement delayed initialization in LLVM mode in a +fairly simple way. + +First, locate a suitable location in the code where the delayed cloning can +take place. This needs to be done with *extreme* care to avoid breaking the +binary. In particular, the program will probably malfunction if you select +a location after: + + - The creation of any vital threads or child processes - since the forkserver + can't clone them easily. + + - The initialization of timers via setitimer() or equivalent calls. + + - The creation of temporary files, network sockets, offset-sensitive file + descriptors, and similar shared-state resources - but only provided that + their state meaningfully influences the behavior of the program later on. + + - Any access to the fuzzed input, including reading the metadata about its + size. + +With the location selected, add this code in the appropriate spot: + +``` +#ifdef __AFL_HAVE_MANUAL_CONTROL + __AFL_INIT(); +#endif +``` + +You don't need the #ifdef guards, but they will make the program still work as +usual when compiled with a tool other than afl-gcc-fast/afl-clang-fast. + +Finally, recompile the program with afl-gcc-fast (afl-gcc or afl-clang will +*not* generate a deferred-initialization binary) - and you should be all set! + +## 5) Bonus feature #2: persistent mode + +Some libraries provide APIs that are stateless, or whose state can be reset in +between processing different input files. When such a reset is performed, a +single long-lived process can be reused to try out multiple test cases, +eliminating the need for repeated fork() calls and the associated OS overhead. + +The basic structure of the program that does this would be: + +``` + while (__AFL_LOOP(1000)) { + + /* Read input data. */ + /* Call library code to be fuzzed. */ + /* Reset state. */ + + } + + /* Exit normally */ +``` + +The numerical value specified within the loop controls the maximum number +of iterations before AFL will restart the process from scratch. This minimizes +the impact of memory leaks and similar glitches; 1000 is a good starting point. + +A more detailed template is shown in ../experimental/persistent_demo/. +Similarly to the previous mode, the feature works only with afl-gcc-fast or +afl-clang-fast; #ifdef guards can be used to suppress it when using other +compilers. + +Note that as with the previous mode, the feature is easy to misuse; if you +do not reset the critical state fully, you may end up with false positives or +waste a whole lot of CPU power doing nothing useful at all. Be particularly +wary of memory leaks and the state of file descriptors. + +When running in this mode, the execution paths will inherently vary a bit +depending on whether the input loop is being entered for the first time or +executed again. To avoid spurious warnings, the feature implies +AFL_NO_VAR_CHECK and hides the "variable path" warnings in the UI. + +PS. Because there are task switches still involved, the mode isn't as fast as +"pure" in-process fuzzing offered, say, by LLVM's LibFuzzer; but it is a lot +faster than the normal fork() model, and compared to in-process fuzzing, +should be a lot more robust. -- cgit 1.4.1 From be6bc155ebd891eebd21eac8982784e7f2be4f41 Mon Sep 17 00:00:00 2001 From: van Hauser Date: Mon, 21 Oct 2019 11:28:32 +0200 Subject: v2.58c --- Makefile | 9 ++++++++- README.md | 18 +++++++++--------- TODO | 5 +---- docs/ChangeLog | 14 ++++++++------ gcc_plugin/README.gcc.md | 4 ---- gcc_plugin/README.whitelist.md | 3 +-- include/config.h | 2 +- 7 files changed, 28 insertions(+), 27 deletions(-) (limited to 'gcc_plugin/README.gcc.md') diff --git a/Makefile b/Makefile index cfcd0890..87c7cdef 100644 --- a/Makefile +++ b/Makefile @@ -104,6 +104,13 @@ man: $(MANPAGES) tests: source-only @cd test ; ./test.sh +performance-tests: performance-test +test-performance: performance-test + +performance-test: source-only + @cd test ; ./test-performance.sh + + help: @echo "HELP --- the following make targets exist:" @echo "==========================================" @@ -307,7 +314,7 @@ install: all $(MANPAGES) install -m 755 $(PROGS) $(SH_PROGS) $${DESTDIR}$(BIN_PATH) rm -f $${DESTDIR}$(BIN_PATH)/afl-as if [ -f afl-qemu-trace ]; then install -m 755 afl-qemu-trace $${DESTDIR}$(BIN_PATH); fi - #if [ -f afl-gcc-fast ]; then set e; install -m 755 afl-gcc-fast $${DESTDIR}$(BIN_PATH); ln -sf afl-gcc-fast $${DESTDIR}$(BIN_PATH)/afl-g++-fast; install -m 755 afl-gcc-pass.so afl-gcc-rt.o $${DESTDIR}$(HELPER_PATH); fi + if [ -f afl-gcc-fast ]; then set e; install -m 755 afl-gcc-fast $${DESTDIR}$(BIN_PATH); ln -sf afl-gcc-fast $${DESTDIR}$(BIN_PATH)/afl-g++-fast; install -m 755 afl-gcc-pass.so afl-gcc-rt.o $${DESTDIR}$(HELPER_PATH); fi ifndef AFL_TRACE_PC if [ -f afl-clang-fast -a -f libLLVMInsTrim.so -a -f afl-llvm-rt.o ]; then set -e; install -m 755 afl-clang-fast $${DESTDIR}$(BIN_PATH); ln -sf afl-clang-fast $${DESTDIR}$(BIN_PATH)/afl-clang-fast++; install -m 755 libLLVMInsTrim.so afl-llvm-pass.so afl-llvm-rt.o $${DESTDIR}$(HELPER_PATH); fi else diff --git a/README.md b/README.md index f0e6faef..4b9537d2 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # american fuzzy lop plus plus (afl++) - Release Version: 2.57c + Release Version: 2.58c - Github Version: 2.57d + Github Version: 2.58d includes all necessary/interesting changes from Google's afl 2.56b @@ -52,13 +52,13 @@ A more thorough list is available in the PATCHES file. - | Feature/Instrumentation | LLVM | GCC | QEMU | Unicorn | - | ----------------------- |:----:|:---:|:----:| -------:| - | laf-intel / CompCov | x | | x | x | - | NeverZero | x(1)| x | x | x | - | Persistent mode | x | | x | | - | Whitelist | x | | | | - | InsTrim | x | | | | + | Feature/Instrumentation | AFL-GCC | LLVM_MODE | GCC_PLUGIN | QEMU_MODE | Unicorn | + | ----------------------- |:-------:|:---------:|:----------:|:---------:|:-------:| + | laf-intel / CompCov | | x | | x | x | + | NeverZero | X | x(1) | | x | x | + | Persistent mode | | x | X | x | | + | Whitelist | | x | X | | | + | InsTrim | | x | | | | (1) only in LLVM >= 9.0 due to a bug in llvm in previous versions diff --git a/TODO b/TODO index 3ca13d6e..f2642b1a 100644 --- a/TODO +++ b/TODO @@ -7,11 +7,8 @@ afl-fuzz: - test the libmutator actually works and does not run infinite (need an example though) gcc_plugin: - - needs to be rewritten - - whitelist support - - skip over uninteresting blocks - - laf-intel - neverZero + - laf-intel qemu_mode: - update to 4.x (probably this will be skipped :( ) diff --git a/docs/ChangeLog b/docs/ChangeLog index 46b9e330..c2d46e4d 100644 --- a/docs/ChangeLog +++ b/docs/ChangeLog @@ -13,14 +13,16 @@ Want to stay in the loop on major new features? Join our mailing list by sending a mail to . ----------------------- -Version ++2.57d (dev): ----------------------- +-------------------------- +Version ++2.58c (release): +-------------------------- - - reverted patch to not unlink and recreate the input file, it resulted in performance loss + - reverted patch to not unlink and recreate the input file, it resulted in + performance loss of ~10% - added test/test-performance.sh script - - (re)added gcc_plugin, fast inline instrumentation is not yet finished - - added gcc_plugin tests + - (re)added gcc_plugin, fast inline instrumentation is not yet finished, + however it includes the whitelisting and persistance feature! by hexcoder- + - gcc_plugin tests added to testing framework -------------------------------- diff --git a/gcc_plugin/README.gcc.md b/gcc_plugin/README.gcc.md index 2e19e911..676ef427 100644 --- a/gcc_plugin/README.gcc.md +++ b/gcc_plugin/README.gcc.md @@ -156,7 +156,3 @@ depending on whether the input loop is being entered for the first time or executed again. To avoid spurious warnings, the feature implies AFL_NO_VAR_CHECK and hides the "variable path" warnings in the UI. -PS. Because there are task switches still involved, the mode isn't as fast as -"pure" in-process fuzzing offered, say, by LLVM's LibFuzzer; but it is a lot -faster than the normal fork() model, and compared to in-process fuzzing, -should be a lot more robust. diff --git a/gcc_plugin/README.whitelist.md b/gcc_plugin/README.whitelist.md index bcc02693..8ad2068d 100644 --- a/gcc_plugin/README.whitelist.md +++ b/gcc_plugin/README.whitelist.md @@ -6,8 +6,7 @@ Using afl++ with partial instrumentation that are interesting to you using the gcc instrumentation provided by afl++. - Originally developed by Christian Holler (:decoder) , - adapted to gcc plugin by hexcoder-. + Plugin by hexcoder-. ## 1) Description and purpose diff --git a/include/config.h b/include/config.h index 2e993a42..f121025d 100644 --- a/include/config.h +++ b/include/config.h @@ -26,7 +26,7 @@ /* Version string: */ -#define VERSION "++2.57d" // c = release, d = volatile github dev +#define VERSION "++2.58c" // c = release, d = volatile github dev /****************************************************** * * -- cgit 1.4.1