Merge pull request #2052 from AFLplusplus/dev v4.20c

4.20 release pre-PR

6 files changed, 102 insertions, 75 deletions

diff --git a/docs/Changelog.md b/docs/Changelog.md
index 48003f4b..2428d63f 100644
--- a/docs/Changelog.md
+++ b/docs/Changelog.md
@@ -3,6 +3,45 @@
   This is the list of all noteworthy changes made in every public
   release of the tool. See README.md for the general instruction manual.
 
+### Version ++4.20c (release)
+  ! A new forkserver communication model is now introduced. afl-fuzz is
+    backward compatible to old compiled targets if they are not built
+    for CMPLOG/Redqueen, but new compiled targets will not work with
+    old afl-fuzz versions!
+  ! Recompile all targets that are instrumented for CMPLOG/Redqueen!
+  - AFL++ now supports up to 4 billion coverage edges, up from 6 million.
+  - New compile option: `make PERFORMANCE=1` - this will enable special
+    CPU dependent optimizations that make everything more performant - but
+    the binaries will likely won't work on different platforms. Also
+    enables a faster hasher if the CPU requirements are met.
+  - The persistent record feature (see config.h) was expanded to also
+    support replay, thanks to @quarta-qti !
+  - afl-fuzz:
+    - the new deterministic fuzzing feature is now activated by default,
+      deactivate with -z. Parameters -d and -D are ignored.
+    - small improvements to CMPLOG/redqueen
+    - workround for a bug with MOpt -L when used with -M - in the future
+      we will either remove or rewrite MOpt.
+    - fix for `-t xxx+` feature
+    - -e extension option now saves the queue items, crashes, etc. with the
+      extension too
+    - fixes for trimmming, correct -V time and reading stats on resume by eqv
+      thanks a lot!
+  - afl-cc:
+    - added collision free caller instrumentation to LTO mode. activate with
+      `AFL_LLVM_LTO_CALLER=1`. You can set a max depth to go through single
+      block functions with `AFL_LLVM_LTO_CALLER_DEPTH` (default 0)
+    - fixes for COMPCOV/LAF and most other modules
+    - fix for GCC_PLUGIN cmplog that broke on std::strings
+  - afl-whatsup:
+    - now also displays current average speed
+    - small bugfixes
+  - Fixes for aflpp custom mutator and standalone tool
+  - Minor edits to afl-persistent-config
+  - Prevent temporary files being left behind on aborted afl-whatsup
+  - More CPU benchmarks added to benchmark/
+
+
 ### Version ++4.10c (release)
   - afl-fuzz:
     - default power schedule is now EXPLORE, due a fix in fast schedules
diff --git a/docs/INSTALL.md b/docs/INSTALL.md
index 84bbe3ea..3089aab2 100644
--- a/docs/INSTALL.md
+++ b/docs/INSTALL.md
@@ -21,7 +21,7 @@ If you want to build AFL++ yourself, you have many options. The easiest choice
 is to build and install everything:
 
 NOTE: depending on your Debian/Ubuntu/Kali/... release, replace `-14` with
-whatever llvm version is available. We recommend llvm 13, 14, 15 or 16.
+whatever llvm version is available. We recommend llvm 13 or newer.
 
 ```shell
 sudo apt-get update
@@ -67,19 +67,20 @@ These build targets exist:
 * unit: perform unit tests (based on cmocka)
 * help: shows these build options
 
-[Unless you are on Mac OS X](https://developer.apple.com/library/archive/qa/qa1118/_index.html),
+[Unless you are on macOS](https://developer.apple.com/library/archive/qa/qa1118/_index.html),
 you can also build statically linked versions of the AFL++ binaries by passing
-the `STATIC=1` argument to make:
+the `PERFORMANCE=1` argument to make:
 
 ```shell
-make STATIC=1
+make PERFORMANCE=1
 ```
 
 These build options exist:
 
-* STATIC - compile AFL++ static
-* CODE_COVERAGE - compile the target for code coverage (see docs/instrumentation/README.llvm.md)
-* ASAN_BUILD - compiles AFL++ with memory sanitizer for debug purposes
+* PERFORMANCE - compile with performance options that make the binary not transferable to other systems. Recommended (except on macOS)!
+* STATIC - compile AFL++ static (does not work on macOS)
+* CODE_COVERAGE - compile the target for code coverage (see [README.llvm.md](../instrumentation/README.llvm.md))
+* ASAN_BUILD - compiles AFL++ with address sanitizer for debug purposes
 * UBSAN_BUILD - compiles AFL++ tools with undefined behaviour sanitizer for debug purposes
 * DEBUG - no optimization, -ggdb3, all warnings and -Werror
 * LLVM_DEBUG - shows llvm deprecation warnings
@@ -91,101 +92,78 @@ These build options exist:
 * NO_NYX - disable building nyx mode dependencies
 * NO_CORESIGHT - disable building coresight (arm64 only)
 * NO_UNICORN_ARM64 - disable building unicorn on arm64
-* AFL_NO_X86 - if compiling on non-intel/amd platforms
+* AFL_NO_X86 - if compiling on non-Intel/AMD platforms
 * LLVM_CONFIG - if your distro doesn't use the standard name for llvm-config (e.g., Debian)
 
 e.g.: `make LLVM_CONFIG=llvm-config-14`
 
-## MacOS X on x86 and arm64 (M1)
+## macOS on x86_64 and arm64
 
-MacOS has some gotchas due to the idiosyncrasies of the platform.
+macOS has some gotchas due to the idiosyncrasies of the platform.
 
-To build AFL, install llvm (and perhaps gcc) from brew and follow the general
-instructions for Linux. If possible, avoid Xcode at all cost.
+macOS supports SYSV shared memory used by AFL++'s instrumentation, but the
+default settings aren't sufficient. Before even building, increase
+them by running the provided script:
+
+```shell
+sudo afl-system-config
+```
+
+See
+[https://www.spy-hill.com/help/apple/SharedMemory.html](https://www.spy-hill.com/help/apple/SharedMemory.html)
+for documentation for the shared memory settings and how to make them permanent.
+
+Next, to build AFL++, install the following packages from brew:
 
 ```shell
 brew install wget git make cmake llvm gdb coreutils
 ```
 
-Be sure to setup `PATH` to point to the correct clang binaries and use the
-freshly installed clang, clang++, llvm-config, gmake and coreutils, e.g.:
+Depending on your macOS system + brew version, brew may be installed in different places.
+You can check with `brew info llvm` to know where, then create a variable for it:
 
 ```shell
-# Depending on your MacOS system + brew version it is either
-export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
-# or
-export PATH="/usr/local/opt/llvm/bin:/usr/local/opt/coreutils/libexec/gnubin:$PATH"
-# you can check with "brew info llvm"
+export HOMEBREW_BASE="/opt/homebrew/opt"
+```
 
-export PATH="/usr/local/bin:$PATH"
+or
+
+```shell
+export HOMEBREW_BASE="/usr/local/opt"
+```
+
+Set `PATH` to point to the brew clang, clang++, llvm-config, gmake and coreutils.
+Also use the brew clang compiler; the Xcode clang compiler must not be used.
+
+```shell
+export PATH="$HOMEBREW_BASE/coreutils/libexec/gnubin:/usr/local/bin:$HOMEBREW_BASE/llvm/bin:$PATH"
 export CC=clang
 export CXX=clang++
-gmake
-cd frida_mode
-gmake
-cd ..
-sudo gmake install
 ```
 
-`afl-gcc` will fail unless you have GCC installed, but that is using outdated
-instrumentation anyway. `afl-clang` might fail too depending on your PATH setup.
-But you don't want neither, you want `afl-clang-fast` anyway :) Note that
-`afl-clang-lto`, `afl-gcc-fast` and `qemu_mode` are not working on MacOS.
+Then build following the general Linux instructions.
 
-The crash reporting daemon that comes by default with MacOS X will cause
-problems with fuzzing. You need to turn it off:
+If everything worked, you should then have `afl-clang-fast` installed, which you can check with:
 
+```shell
+which afl-clang-fast
 ```
-launchctl unload -w /System/Library/LaunchAgents/com.apple.ReportCrash.plist
-sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.ReportCrash.Root.plist
-```
 
-The `fork()` semantics on OS X are a bit unusual compared to other unix systems
+Note that `afl-clang-lto`, `afl-gcc-fast` and `qemu_mode` are not working on macOS.
+
+The crash reporting daemon that comes by default with macOS will cause
+problems with fuzzing. You need to turn it off, which you can do with `afl-system-config`.
+
+The `fork()` semantics on macOS are a bit unusual compared to other unix systems
 and definitely don't look POSIX-compliant. This means two things:
 
   - Fuzzing will be probably slower than on Linux. In fact, some folks report
     considerable performance gains by running the jobs inside a Linux VM on
-    MacOS X.
+    macOS.
   - Some non-portable, platform-specific code may be incompatible with the AFL++
     forkserver. If you run into any problems, set `AFL_NO_FORKSRV=1` in the
     environment before starting afl-fuzz.
 
-User emulation mode of QEMU does not appear to be supported on MacOS X, so
+User emulation mode of QEMU does not appear to be supported on macOS, so
 black-box instrumentation mode (`-Q`) will not work. However, FRIDA mode (`-O`)
-works on both x86 and arm64 MacOS boxes.
-
-MacOS X supports SYSV shared memory used by AFL's instrumentation, but the
-default settings aren't usable with AFL++. The default settings on 10.14 seem to
-be:
-
-```bash
-$ ipcs -M
-IPC status from <running system> as of XXX
-shminfo:
-        shmmax: 4194304 (max shared memory segment size)
-        shmmin:       1 (min shared memory segment size)
-        shmmni:      32 (max number of shared memory identifiers)
-        shmseg:       8 (max shared memory segments per process)
-        shmall:    1024 (max amount of shared memory in pages)
-```
-
-To temporarily change your settings to something minimally usable with AFL++,
-run these commands as root:
-
-```bash
-sysctl kern.sysv.shmmax=8388608
-sysctl kern.sysv.shmall=4096
-```
-
-If you're running more than one instance of AFL, you likely want to make
-`shmall` bigger and increase `shmseg` as well:
-
-```bash
-sysctl kern.sysv.shmmax=8388608
-sysctl kern.sysv.shmseg=48
-sysctl kern.sysv.shmall=98304
-```
-
-See
-[http://www.spy-hill.com/help/apple/SharedMemory.html](http://www.spy-hill.com/help/apple/SharedMemory.html)
-for documentation for these settings and how to make them permanent.
+works on both x86 and arm64 macOS boxes.
diff --git a/docs/env_variables.md b/docs/env_variables.md
index a972b6da..1e4fc7ba 100644
--- a/docs/env_variables.md
+++ b/docs/env_variables.md
@@ -248,6 +248,9 @@ use (which only ever the author of this LTO implementation will use). These are
 used if several separated instrumentations are performed which are then later
 combined.
 
+  - `AFL_LLVM_LTO_CALLER` activates collision free CALLER instrumentation
+  - `AFL_LLVM_LTO_CALLER` sets the maximum mumber of single block functions
+    to dig deeper into a real function. Default 0.
   - `AFL_LLVM_DOCUMENT_IDS=file` will document to a file which edge ID was given
     to which function. This helps to identify functions with variable bytes or
     which functions were touched by an input.
diff --git a/docs/fuzzing_in_depth.md b/docs/fuzzing_in_depth.md
index 6a217641..82437807 100644
--- a/docs/fuzzing_in_depth.md
+++ b/docs/fuzzing_in_depth.md
@@ -958,7 +958,7 @@ too long for your overall available fuzz run time.
    campaign but not good for short CI runs.
 
 How this can look like can, e.g., be seen at AFL++'s setup in Google's
-[oss-fuzz](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/compile_afl)
+[previous oss-fuzz version](https://github.com/google/oss-fuzz/blob/3e2c5312417d1a6f9564472f3df1fd27759b289d/infra/base-images/base-builder/compile_afl)
 and
 [clusterfuzz](https://github.com/google/clusterfuzz/blob/master/src/clusterfuzz/_internal/bot/fuzzers/afl/launcher.py).
 
diff --git a/docs/third_party_tools.md b/docs/third_party_tools.md
index 02a40ce5..64a37c83 100644
--- a/docs/third_party_tools.md
+++ b/docs/third_party_tools.md
@@ -5,6 +5,10 @@
 * [afl-rs](https://github.com/rust-fuzz/afl.rs) - AFL++ for RUST
 * [WASM](https://github.com/fgsect/WAFL) - AFL++ for WASM
 
+## Starting multiple AFL++ instances in parallel with recommended settings:
+* [https://github.com/0xricksanchez/AFL_Runner](https://github.com/0xricksanchez/AFL_Runner)
+* [https://github.com/MegaManSec/AFLplusplus-Parallel-Gen](https://github.com/MegaManSec/AFLplusplus-Parallel-Gen)
+
 ## Speeding up fuzzing
 
 * [libfiowrapper](https://github.com/marekzmyslowski/libfiowrapper) - if the
diff --git a/docs/tutorials.md b/docs/tutorials.md
index 0a09f6dc..f9378232 100644
--- a/docs/tutorials.md
+++ b/docs/tutorials.md
@@ -21,6 +21,9 @@ training, then we can highly recommend the following:
 
 * [https://github.com/antonio-morales/Fuzzing101](https://github.com/antonio-morales/Fuzzing101)
 
+A good workflow overview (like our [fuzzing_in_depth.md](fuzzing_in_depth.md)):
+* [https://appsec.guide/docs/fuzzing/c-cpp/aflpp/](https://appsec.guide/docs/fuzzing/c-cpp/aflpp/)
+
 Here is a good workflow description (and tutorial) for qemu_mode:
 
 * [https://airbus-seclab.github.io/AFLplusplus-blogpost/](https://airbus-seclab.github.io/AFLplusplus-blogpost/)