Merge pull request #1759 from AFLplusplus/dev

Dev

8 files changed, 135 insertions, 29 deletions

diff --git a/docs/Changelog.md b/docs/Changelog.md
index 3c06a785..facf2196 100644
--- a/docs/Changelog.md
+++ b/docs/Changelog.md
@@ -3,24 +3,64 @@
   This is the list of all noteworthy changes made in every public
   release of the tool. See README.md for the general instruction manual.
 
-### Version ++4.06a (dev)
+### Version ++4.07a (dev)
+  - afl-fuzz:
+    - reverse reading the seeds only on restarts (increases performance)
+    - new env `AFL_POST_PROCESS_KEEP_ORIGINAL` to keep the orignal
+      data before post process on finds (for atnwalk custom mutator)
+    - new env `AFL_IGNORE_PROBLEMS_COVERAGE` to ignore coverage from
+      loaded libs after forkserver initialization (required by Mozilla)
+  - afl-cc:
+    - added @responsefile support
+    - new env `AFL_LLVM_LTO_SKIPINIT` to support the AFL++ based WASM
+      (https://github.com/fgsect/WAFL) project
+    - error and print help if afl-clan-lto is used with lto=thin
+    - rewrote our PCGUARD pass to be compatible with LLVM 15+ shenanigans,
+      requires LLVM 13+ now instead of 10.0.1+
+    - fallback to native LLVM PCGUARD if our PCGUARD is unavailable
+  - afl-showmap:
+    - added custom mutator post_process and send support
+    - add `-I filelist` option, an alternative to `-i in_dir`
+  - afl-cmin + afl-cmin.bash:
+    - `-T threads` parallel task support, can be a huge speedup!
+  - qemu_mode:
+    - Persistent mode + QASAN support for ppc32 targets by @worksbutnottested
+  - a new grammar custom mutator atnwalk was submitted by @voidptr127 !
+  - two new custom mutators are now available:
+    - TritonDSE in custom_mutators/aflpp_tritondse
+    - SymQEMU in custom_mutators/symqemu
+
+
+### Version ++4.06c (release)
   - afl-fuzz:
     - ensure temporary file descriptor is closed when not used
     - added `AFL_NO_WARN_INSTABILITY`
-    - added `AFL_FRIDA_STATS_INTERVAL`
     - added time_wo_finds to fuzzer_stats
+    - fixed a crash in pizza (1st april easter egg) mode. Sorry for
+      everyone who was affected!
+    - allow pizza mode to be disabled when AFL_PIZZA_MODE is set to -1
+    - option `-p mmopt` now also selects new queue items more often
+    - fix bug in post_process custom mutator implementation
+    - print name of custom mutator in UI
+    - slight changes that improve fuzzer performance
   - afl-cc:
     - add CFI sanitizer variant to gcc targets
-    - llvm 16 support (thanks to @devnexen!)
+    - llvm 16 + 17 support (thanks to @devnexen!)
     - support llvm 15 native pcguard changes
     - support for LLVMFuzzerTestOneInput -1 return
+    - LTO autoken and llvm_mode: added AFL_LLVM_DICT2FILE_NO_MAIN support
   - qemu_mode:
     - fix _RANGES envs to allow hyphens in the filenames
-  - new custom module: autotoken, grammar free fuzzer for text inputs
-  - LTO autoken and llvm_mode: added AFL_LLVM_DICT2FILE_NO_MAIN support
+    - basic riscv support
+  - frida_mode:
+    - added `AFL_FRIDA_STATS_INTERVAL`
+    - fix issue on MacOS
+  - unicorn_mode:
+    - updated and minor issues fixed
+  - nyx_mode support for all tools
   - better sanitizer default options support for all tools
-  - unicorn_mode: updated and minor issues fixed
-  - frida_mode: fix issue on MacOS
+  - new custom module: autotoken, a grammar free fuzzer for text inputs
+  - fixed custom mutator C examples
   - more minor fixes and cross-platform support
 
 ### Version ++4.05c (release)
@@ -199,7 +239,7 @@
     afl-showmap and other tools.
   - afl-cc:
     - detect overflow reads on initial input buffer for asan
-    - new cmplog mode (incompatible with older afl++ versions)
+    - new cmplog mode (incompatible with older AFL++ versions)
     - support llvm IR select instrumentation for default PCGUARD and LTO
     - fix for shared linking on MacOS
     - better selective instrumentation AFL_LLVM_{ALLOW|DENY}LIST
diff --git a/docs/FAQ.md b/docs/FAQ.md
index 4a9080f8..8178db46 100644
--- a/docs/FAQ.md
+++ b/docs/FAQ.md
@@ -171,6 +171,14 @@ If you find an interesting or important question missing, submit it via
   The more "unstable" edges there are, the harder it is for AFL++ to identify
   valid new paths.
 
+  If you fuzz in persistent mode (`AFL_LOOP` or `LLVMFuzzerTestOneInput()`
+  harnesses, a large number of unstable edges can mean that the target keeps
+  internal state and therefore it is possible that crashes cannot be replayed.
+  In such a case do either **not** fuzz in persistent mode (remove `AFL_LOOP()`
+  from your harness or call `LLVMFuzzerTestOneInput()` harnesses with `@@`),
+  or set a low  `AFL_LOOP` value, e.g. 100, and enable `AFL_PERSISTENT_RECORD`
+  in `config.h` with the same value.
+
   A value above 90% is usually fine and a value above 80% is also still ok, and
   even a value above 20% can still result in successful finds of bugs. However,
   it is recommended that for values below 90% or 80% you should take
@@ -229,7 +237,8 @@ If you find an interesting or important question missing, submit it via
   If this is not a viable option, you can set `AFL_IGNORE_PROBLEMS=1` but then
   the existing map will be used also for the newly loaded libraries, which
   allows it to work, however, the efficiency of the fuzzing will be partially
-  degraded.
+  degraded. Note that there is additionally `AFL_IGNORE_PROBLEMS_COVERAGE` to
+  additionally tell AFL++ to ignore any coverage from the late loaded libaries.
 </p></details>
 
 <details>
diff --git a/docs/INSTALL.md b/docs/INSTALL.md
index 591b7ded..9005a7eb 100644
--- a/docs/INSTALL.md
+++ b/docs/INSTALL.md
@@ -3,9 +3,8 @@
 ## Linux on x86
 
 An easy way to install AFL++ with everything compiled is available via docker:
-You can use the [Dockerfile](../Dockerfile) (which has gcc-10 and clang-12 -
-hence afl-clang-lto is available) or just pull directly from the Docker Hub
-(for x86_64 and arm64):
+You can use the [Dockerfile](../Dockerfile) or just pull directly from the
+Docker Hub (for x86_64 and arm64):
 
 ```shell
 docker pull aflplusplus/aflplusplus:
@@ -21,14 +20,14 @@ development state of AFL++.
 If you want to build AFL++ yourself, you have many options. The easiest choice
 is to build and install everything:
 
-NOTE: depending on your Debian/Ubuntu/Kali/... release, replace `-12` with
-whatever llvm version is available. We recommend llvm 12, 13 or 14.
+NOTE: depending on your Debian/Ubuntu/Kali/... release, replace `-14` with
+whatever llvm version is available. We recommend llvm 13, 14, 15 or 16.
 
 ```shell
 sudo apt-get update
 sudo apt-get install -y build-essential python3-dev automake cmake git flex bison libglib2.0-dev libpixman-1-dev python3-setuptools cargo libgtk-3-dev
-# try to install llvm 12 and install the distro default if that fails
-sudo apt-get install -y lld-12 llvm-12 llvm-12-dev clang-12 || sudo apt-get install -y lld llvm llvm-dev clang
+# try to install llvm 14 and install the distro default if that fails
+sudo apt-get install -y lld-14 llvm-14 llvm-14-dev clang-14 || sudo apt-get install -y lld llvm llvm-dev clang
 sudo apt-get install -y gcc-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-plugin-dev libstdc++-$(gcc --version|head -n1|sed 's/\..*//'|sed 's/.* //')-dev
 sudo apt-get install -y ninja-build # for QEMU mode
 git clone https://github.com/AFLplusplus/AFLplusplus
@@ -51,7 +50,7 @@ make source-only
 
 These build targets exist:
 
-* all: the main afl++ binaries and llvm/gcc instrumentation
+* all: the main AFL++ binaries and llvm/gcc instrumentation
 * binary-only: everything for binary-only fuzzing: frida_mode, nyx_mode,
   qemu_mode, frida_mode, unicorn_mode, coresight_mode, libdislocator,
   libtokencap
@@ -79,22 +78,20 @@ make STATIC=1
 These build options exist:
 
 * STATIC - compile AFL++ static
+* CODE_COVERAGE - compile the target for code coverage (see docs/instrumentation/README.llvm.md)
 * ASAN_BUILD - compiles AFL++ with memory sanitizer for debug purposes
-* UBSAN_BUILD - compiles AFL++ tools with undefined behaviour sanitizer for
-  debug purposes
+* UBSAN_BUILD - compiles AFL++ tools with undefined behaviour sanitizer for debug purposes
 * DEBUG - no optimization, -ggdb3, all warnings and -Werror
 * LLVM_DEBUG - shows llvm deprecation warnings
 * PROFILING - compile afl-fuzz with profiling information
 * INTROSPECTION - compile afl-fuzz with mutation introspection
 * NO_PYTHON - disable python support
-* NO_SPLICING - disables splicing mutation in afl-fuzz, not recommended for
-  normal fuzzing
+* NO_SPLICING - disables splicing mutation in afl-fuzz, not recommended for normal fuzzing
 * NO_NYX - disable building nyx mode dependencies
 * NO_CORESIGHT - disable building coresight (arm64 only)
 * NO_UNICORN_ARM64 - disable building unicorn on arm64
 * AFL_NO_X86 - if compiling on non-intel/amd platforms
-* LLVM_CONFIG - if your distro doesn't use the standard name for llvm-config
-  (e.g., Debian)
+* LLVM_CONFIG - if your distro doesn't use the standard name for llvm-config (e.g., Debian)
 
 e.g.: `make LLVM_CONFIG=llvm-config-14`
 
diff --git a/docs/best_practices.md b/docs/best_practices.md
index 133c645e..459fcaf7 100644
--- a/docs/best_practices.md
+++ b/docs/best_practices.md
@@ -131,6 +131,11 @@ jitter, or is a hash map function etc., then it should not be instrumented.
 To be able to exclude these functions (based on AFL++'s measured stability), the
 following process will allow to identify functions with variable edges.
 
+Note that this is only useful for non-persistent targets!
+If a persistent target is unstable whereas when run non-persistent is fine,
+then this means that the target is keeping internal state, which is bad for
+fuzzing. Fuzz such targets **without** persistent mode.
+
 Four steps are required to do this and it also requires quite some knowledge of
 coding and/or disassembly and is effectively possible only with `afl-clang-fast`
 `PCGUARD` and `afl-clang-lto` `LTO` instrumentation.
diff --git a/docs/custom_mutators.md b/docs/custom_mutators.md
index 82131c92..3f7e9e6e 100644
--- a/docs/custom_mutators.md
+++ b/docs/custom_mutators.md
@@ -118,7 +118,7 @@ def deinit():  # optional for Python
 
 ### Custom Mutation
 
-- `init`:
+- `init` (optional in Python):
 
     This method is called when AFL++ starts up and is used to seed RNG and set
     up buffers and state.
@@ -184,6 +184,11 @@ def deinit():  # optional for Python
     to the target, e.g. if it is too short, too corrupted, etc. If so,
     return a NULL buffer and zero length (or a 0 length string in Python).
 
+    NOTE: Do not make any random changes to the data in this function!
+
+    PERFORMANCE for C/C++: If possible make the changes in-place (so modify
+    the `*data` directly, and return it as `*outbuf = data`.
+
 - `fuzz_send` (optional):
 
     This method can be used if you want to send data to the target yourself,
@@ -202,7 +207,7 @@ def deinit():  # optional for Python
     discovered if compiled with INTROSPECTION. The custom mutator can then
     return a string (const char *) that reports the exact mutations used.
 
-- `deinit`:
+- `deinit` (optional in Python):
 
     The last method to be called, deinitializing the state.
 
@@ -299,6 +304,34 @@ Note: for some distributions, you might also need the package `python[3]-apt`.
 In case your setup is different, set the necessary variables like this:
 `PYTHON_INCLUDE=/path/to/python/include LDFLAGS=-L/path/to/python/lib make`.
 
+### Helpers
+
+For C/C++ custom mutators you get a pointer to `afl_state_t *afl` in the
+`afl_custom_init()` which contains all information that you need.
+Note that if you access it, you need to recompile your custom mutator if
+you update AFL++ because the structure might have changed!
+
+For mutators written in Python, Rust, GO, etc. there are a few environment
+variables set to help you to get started:
+
+`AFL_CUSTOM_INFO_PROGRAM` - the program name of the target that is executed.
+If your custom mutator is used with modes like Qemu (`-Q`), this will still
+contain the target program, not afl-qemu-trace.
+
+`AFL_CUSTOM_INFO_PROGRAM_INPUT` - if the `-f` parameter is used with afl-fuzz
+then this value is found in this environment variable.
+
+`AFL_CUSTOM_INFO_PROGRAM_ARGV` - this contains the parameters given to the
+target program and still has the `@@` identifier in there.
+
+Note: If `AFL_CUSTOM_INFO_PROGRAM_INPUT` is empty and `AFL_CUSTOM_INFO_PROGRAM_ARGV`
+is either empty or does not contain `@@` then the target gets the input via
+`stdin`.
+
+`AFL_CUSTOM_INFO_OUT` - This is the output directory for this fuzzer instance,
+so if `afl-fuzz` was called with `-o out -S foobar`, then this will be set to
+`out/foobar`.
+
 ### Custom Mutator Preparation
 
 For C/C++ mutators, the source code must be compiled as a shared object:
diff --git a/docs/env_variables.md b/docs/env_variables.md
index c9dc1bbd..0f0869d2 100644
--- a/docs/env_variables.md
+++ b/docs/env_variables.md
@@ -156,7 +156,7 @@ Available options:
   - LTO - LTO instrumentation
   - NATIVE - clang's original pcguard based instrumentation
   - NGRAM-x - deeper previous location coverage (from NGRAM-2 up to NGRAM-16)
-  - PCGUARD - our own pcgard based instrumentation (default)
+  - PCGUARD - our own pcguard based instrumentation (default)
 
 #### CMPLOG
 
@@ -240,7 +240,9 @@ combined.
     the default `0x10000`. A value of 0 or empty sets the map address to be
     dynamic (the original AFL way, which is slower).
   - `AFL_LLVM_MAP_DYNAMIC` sets the shared memory address to be dynamic.
-
+  - `AFL_LLVM_LTO_SKIPINIT` skips adding initialization code. Some global vars
+    (e.g. the highest location ID) are not injected. Needed to instrument with
+    [WAFL](https://github.com/fgsect/WAFL.git).
   For more information, see
   [instrumentation/README.lto.md](../instrumentation/README.lto.md).
 
@@ -404,7 +406,8 @@ checks or alter some of the more exotic semantics of the tool:
 
   - If afl-fuzz encounters an incorrect fuzzing setup during a fuzzing session
     (not at startup), it will terminate. If you do not want this, then you can
-    set `AFL_IGNORE_PROBLEMS`.
+    set `AFL_IGNORE_PROBLEMS`. If you additionally want to also ignore coverage
+    from late loaded libraries, you can set `AFL_IGNORE_PROBLEMS_COVERAGE`.
 
   - When running in the `-M` or `-S` mode, setting `AFL_IMPORT_FIRST` causes the
     fuzzer to import test cases from other instances before doing anything else.
@@ -581,7 +584,7 @@ checks or alter some of the more exotic semantics of the tool:
     constructors in your target, you can set `AFL_EARLY_FORKSERVER`.
     Note that this is not a compile time option but a runtime option :-)
 
-  - Set `AFL_PIZZA_MODE` to 1 to enable the April 1st stats menu, set to 0
+  - Set `AFL_PIZZA_MODE` to 1 to enable the April 1st stats menu, set to -1
     to disable although it is 1st of April.
 
   - If you need a specific interval to update fuzzer_stats file, you can
@@ -616,6 +619,14 @@ The QEMU wrapper used to instrument binary-only code supports several settings:
   - Setting `AFL_INST_LIBS` causes the translator to also instrument the code
     inside any dynamically linked libraries (notably including glibc).
 
+  - You can use `AFL_QEMU_INST_RANGES=0xaaaa-0xbbbb,0xcccc-0xdddd` to just
+    instrument specific memory locations, e.g. a specific library.
+    Excluding ranges takes priority over any included ranges or `AFL_INST_LIBS`.
+
+  - You can use `AFL_QEMU_EXCLUDE_RANGES=0xaaaa-0xbbbb,0xcccc-0xdddd` to **NOT**
+    instrument specific memory locations, e.g. a specific library.
+    Excluding ranges takes priority over any included ranges or `AFL_INST_LIBS`.
+
   - It is possible to set `AFL_INST_RATIO` to skip the instrumentation on some
     of the basic blocks, which can be useful when dealing with very complex
     binaries.
@@ -677,6 +688,8 @@ support.
 * `AFL_FRIDA_INST_JIT` - Enable the instrumentation of Just-In-Time compiled
   code. Code is considered to be JIT if the executable segment is not backed by
   a file.
+* `AFL_FRIDA_INST_NO_DYNAMIC_LOAD` - Don't instrument the code loaded late at
+  runtime. Strictly limits instrumentation to what has been included.
 * `AFL_FRIDA_INST_NO_OPTIMIZE` - Don't use optimized inline assembly coverage
   instrumentation (the default where available). Required to use
   `AFL_FRIDA_INST_TRACE`.
diff --git a/docs/third_party_tools.md b/docs/third_party_tools.md
index 97f2c362..02a40ce5 100644
--- a/docs/third_party_tools.md
+++ b/docs/third_party_tools.md
@@ -1,5 +1,10 @@
 # Tools that help fuzzing with AFL++
 
+## AFL++ and other development languages
+
+* [afl-rs](https://github.com/rust-fuzz/afl.rs) - AFL++ for RUST
+* [WASM](https://github.com/fgsect/WAFL) - AFL++ for WASM
+
 ## Speeding up fuzzing
 
 * [libfiowrapper](https://github.com/marekzmyslowski/libfiowrapper) - if the
diff --git a/docs/tutorials.md b/docs/tutorials.md
index 758fddab..342080fd 100644
--- a/docs/tutorials.md
+++ b/docs/tutorials.md
@@ -20,6 +20,10 @@ training, then we can highly recommend the following:
 
 * [https://github.com/antonio-morales/Fuzzing101](https://github.com/antonio-morales/Fuzzing101)
 
+Here is good workflow description for frida_mode:
+
+* [https://blog.quarkslab.com/android-greybox-fuzzing-with-afl-frida-mode.html](https://blog.quarkslab.com/android-greybox-fuzzing-with-afl-frida-mode.html)
+
 If you are interested in fuzzing structured data (where you define what the
 structure is), these links have you covered (some are outdated though):