Merge pull request #1 from AFLplusplus/stable

sync

9 files changed, 141 insertions, 33 deletions

diff --git a/docs/Changelog.md b/docs/Changelog.md
index c5eb6be3..20b915fa 100644
--- a/docs/Changelog.md
+++ b/docs/Changelog.md
@@ -3,15 +3,75 @@
   This is the list of all noteworthy changes made in every public
   release of the tool. See README.md for the general instruction manual.
 
-### Version ++4.05a (dev)
+### Version ++4.07a (dev)
+  - afl-fuzz:
+    - reverse reading the seeds only on restarts (increases performance)
+    - new env `AFL_POST_PROCESS_KEEP_ORIGINAL` to keep the orignal
+      data before post process on finds (for atnwalk custom mutator)
+    - new env `AFL_IGNORE_PROBLEMS_COVERAGE` to ignore coverage from
+      loaded libs after forkserver initialization (required by Mozilla)
+  - afl-cc:
+    - new env `AFL_LLVM_LTO_SKIPINIT` to support the AFL++ based WASM
+      (https://github.com/fgsect/WAFL) project
+  - afl-showmap:
+    - added custom mutator post_process and send support
+    - add `-I filelist` option, an alternative to `-i in_dir`
+  - afl-cmin + afl-cmin.bash:
+    - `-T threads` parallel task support, can be a huge speedup!
+  - a new grammar custom mutator atnwalk was submitted by @voidptr127 !
+
+
+### Version ++4.06c (release)
+  - afl-fuzz:
+    - ensure temporary file descriptor is closed when not used
+    - added `AFL_NO_WARN_INSTABILITY`
+    - added time_wo_finds to fuzzer_stats
+    - fixed a crash in pizza (1st april easter egg) mode. Sorry for
+      everyone who was affected!
+    - allow pizza mode to be disabled when AFL_PIZZA_MODE is set to -1
+    - option `-p mmopt` now also selects new queue items more often
+    - fix bug in post_process custom mutator implementation
+    - print name of custom mutator in UI
+    - slight changes that improve fuzzer performance
+  - afl-cc:
+    - add CFI sanitizer variant to gcc targets
+    - llvm 16 + 17 support (thanks to @devnexen!)
+    - support llvm 15 native pcguard changes
+    - support for LLVMFuzzerTestOneInput -1 return
+    - LTO autoken and llvm_mode: added AFL_LLVM_DICT2FILE_NO_MAIN support
+  - qemu_mode:
+    - fix _RANGES envs to allow hyphens in the filenames
+    - basic riscv support
+  - frida_mode:
+    - added `AFL_FRIDA_STATS_INTERVAL`
+    - fix issue on MacOS
+  - unicorn_mode:
+    - updated and minor issues fixed
+  - nyx_mode support for all tools
+  - better sanitizer default options support for all tools
+  - new custom module: autotoken, a grammar free fuzzer for text inputs
+  - fixed custom mutator C examples
+  - more minor fixes and cross-platform support
+
+### Version ++4.05c (release)
+  - MacOS: libdislocator, libtokencap etc. do not work with modern
+    MacOS anymore, but could be patched to work, see this issue if you
+    want to make the effort and send a PR:
+    https://github.com/AFLplusplus/AFLplusplus/issues/1594
   - afl-fuzz:
     - added afl_custom_fuzz_send custom mutator feature. Now your can
       send fuzz data to the target as you need, e.g. via IPC.
-    - cmplog mode now has -l R option for random colorization, thanks
+    - cmplog mode now has a -l R option for random colorization, thanks
       to guyf2010 for the PR!
+    - queue statistics are written every 30 minutes to
+      out/NAME/queue_data if compiled with INTROSPECTION
+    - new env: AFL_FORK_SERVER_KILL_SIGNAL
   - afl-showmap/afl-cmin
-    - -t none now translates to -t 120000 (120 seconds)
+    - `-t none` now translates to `-t 120000` (120 seconds)
   - unicorn_mode updated
+  - updated rust custom mutator dependencies and LibAFL custom mutator
+  - overall better sanitizer default setting handling
+  - several minor bugfixes
 
 ### Version ++4.04c (release)
   - fix gramatron and grammar_mutator build scripts
diff --git a/docs/FAQ.md b/docs/FAQ.md
index 4a9080f8..76350c79 100644
--- a/docs/FAQ.md
+++ b/docs/FAQ.md
@@ -229,7 +229,8 @@ If you find an interesting or important question missing, submit it via
   If this is not a viable option, you can set `AFL_IGNORE_PROBLEMS=1` but then
   the existing map will be used also for the newly loaded libraries, which
   allows it to work, however, the efficiency of the fuzzing will be partially
-  degraded.
+  degraded. Note that there is additionally `AFL_IGNORE_PROBLEMS_COVERAGE` to
+  additionally tell AFL++ to ignore any coverage from the late loaded libaries.
 </p></details>
 
 <details>
diff --git a/docs/afl-fuzz_approach.md b/docs/afl-fuzz_approach.md
index 6af39769..cb173f10 100644
--- a/docs/afl-fuzz_approach.md
+++ b/docs/afl-fuzz_approach.md
@@ -483,6 +483,7 @@ directory. This includes:
 - `fuzzer_pid`        - PID of the fuzzer process
 - `cycles_done`       - queue cycles completed so far
 - `cycles_wo_finds`   - number of cycles without any new paths found
+- `time_wo_finds`     - longest time in seconds no new path was found
 - `execs_done`        - number of execve() calls attempted
 - `execs_per_sec`     - overall number of execs per second
 - `corpus_count`      - total number of entries in the queue
diff --git a/docs/custom_mutators.md b/docs/custom_mutators.md
index 4ffeda7a..a1de479e 100644
--- a/docs/custom_mutators.md
+++ b/docs/custom_mutators.md
@@ -48,6 +48,7 @@ C/C++:
 ```c
 void *afl_custom_init(afl_state_t *afl, unsigned int seed);
 unsigned int afl_custom_fuzz_count(void *data, const unsigned char *buf, size_t buf_size);
+void afl_custom_splice_optout(void *data);
 size_t afl_custom_fuzz(void *data, unsigned char *buf, size_t buf_size, unsigned char **out_buf, unsigned char *add_buf, size_t add_buf_size, size_t max_size);
 const char *afl_custom_describe(void *data, size_t max_description_len);
 size_t afl_custom_post_process(void *data, unsigned char *buf, size_t buf_size, unsigned char **out_buf);
@@ -72,6 +73,9 @@ def init(seed):
 def fuzz_count(buf):
     return cnt
 
+def splice_optout()
+    pass
+
 def fuzz(buf, add_buf, max_size):
     return mutated_out
 
@@ -114,7 +118,7 @@ def deinit():  # optional for Python
 
 ### Custom Mutation
 
-- `init`:
+- `init` (optional in Python):
 
     This method is called when AFL++ starts up and is used to seed RNG and set
     up buffers and state.
@@ -132,6 +136,13 @@ def deinit():  # optional for Python
     for a specific queue entry, use this function. This function is most useful
     if `AFL_CUSTOM_MUTATOR_ONLY` is **not** used.
 
+- `splice_optout` (optional):
+
+    If this function is present, no splicing target is passed to the `fuzz`
+    function. This saves time if splicing data is not needed by the custom
+    fuzzing function.
+    This function is never called, just needs to be present to activate.
+
 - `fuzz` (optional):
 
     This method performs custom mutations on a given input. It also accepts an
@@ -139,6 +150,7 @@ def deinit():  # optional for Python
     sense to use it. You would only skip this if `post_process` is used to fix
     checksums etc. so if you are using it, e.g., as a post processing library.
     Note that a length > 0 *must* be returned!
+    The returned output buffer is under **your** memory management!
 
 - `describe` (optional):
 
@@ -172,6 +184,11 @@ def deinit():  # optional for Python
     to the target, e.g. if it is too short, too corrupted, etc. If so,
     return a NULL buffer and zero length (or a 0 length string in Python).
 
+    NOTE: Do not make any random changes to the data in this function!
+
+    PERFORMANCE for C/C++: If possible make the changes in-place (so modify
+    the `*data` directly, and return it as `*outbuf = data`.
+
 - `fuzz_send` (optional):
 
     This method can be used if you want to send data to the target yourself,
@@ -190,7 +207,7 @@ def deinit():  # optional for Python
     discovered if compiled with INTROSPECTION. The custom mutator can then
     return a string (const char *) that reports the exact mutations used.
 
-- `deinit`:
+- `deinit` (optional in Python):
 
     The last method to be called, deinitializing the state.
 
diff --git a/docs/env_variables.md b/docs/env_variables.md
index d1c13e15..b1f23159 100644
--- a/docs/env_variables.md
+++ b/docs/env_variables.md
@@ -129,6 +129,9 @@ subset of the settings discussed in section 1, with the exception of:
     write all constant string comparisons to this file to be used later with
     afl-fuzz' `-x` option.
 
+  - An option to `AFL_LLVM_DICT2FILE` is `AFL_LLVM_DICT2FILE_NO_MAIN=1` which
+    skill not parse `main()`.
+
   - `TMPDIR` and `AFL_KEEP_ASSEMBLY`, since no temporary assembly files are
     created.
 
@@ -153,7 +156,7 @@ Available options:
   - LTO - LTO instrumentation
   - NATIVE - clang's original pcguard based instrumentation
   - NGRAM-x - deeper previous location coverage (from NGRAM-2 up to NGRAM-16)
-  - PCGUARD - our own pcgard based instrumentation (default)
+  - PCGUARD - our own pcguard based instrumentation (default)
 
 #### CMPLOG
 
@@ -237,7 +240,9 @@ combined.
     the default `0x10000`. A value of 0 or empty sets the map address to be
     dynamic (the original AFL way, which is slower).
   - `AFL_LLVM_MAP_DYNAMIC` sets the shared memory address to be dynamic.
-
+  - `AFL_LLVM_LTO_SKIPINIT` skips adding initialization code. Some global vars
+    (e.g. the highest location ID) are not injected. Needed to instrument with
+    [WAFL](https://github.com/fgsect/WAFL.git).
   For more information, see
   [instrumentation/README.lto.md](../instrumentation/README.lto.md).
 
@@ -354,6 +359,9 @@ checks or alter some of the more exotic semantics of the tool:
   - Setting `AFL_KEEP_TIMEOUTS` will keep longer running inputs if they reach
     new coverage
 
+  - On the contrary, if you are not interested in any timeouts, you can set
+    `AFL_IGNORE_TIMEOUTS` to get a bit of speed instead.
+
   - `AFL_EXIT_ON_SEED_ISSUES` will restore the vanilla afl-fuzz behavior which
     does not allow crashes or timeout seeds in the initial -i corpus.
 
@@ -378,10 +386,10 @@ checks or alter some of the more exotic semantics of the tool:
     valid terminal was detected (for virtual consoles).
 
   - Setting `AFL_FORKSRV_INIT_TMOUT` allows you to specify a different timeout
-    to wait for the forkserver to spin up. The default is the `-t` value times
-    `FORK_WAIT_MULT` from `config.h` (usually 10), so for a `-t 100`, the
-    default would wait for `1000` milliseconds. Setting a different time here is
-    useful if the target has a very slow startup time, for example, when doing
+    to wait for the forkserver to spin up. The specified value is the new timeout, in milliseconds.
+    The default is the `-t` value times `FORK_WAIT_MULT` from `config.h` (usually 10), so for a `-t 100`, the default would wait for `1000` milliseconds.
+    The `AFL_FORKSRV_INIT_TMOUT` value does not get multiplied. It overwrites the initial timeout afl-fuzz waits for the target to come up with a constant time.
+    Setting a different time here is useful if the target has a very slow startup time, for example, when doing
     full-system fuzzing or emulation, but you don't want the actual runs to wait
     too long for timeouts.
 
@@ -398,7 +406,8 @@ checks or alter some of the more exotic semantics of the tool:
 
   - If afl-fuzz encounters an incorrect fuzzing setup during a fuzzing session
     (not at startup), it will terminate. If you do not want this, then you can
-    set `AFL_IGNORE_PROBLEMS`.
+    set `AFL_IGNORE_PROBLEMS`. If you additionally want to also ignore coverage
+    from late loaded libraries, you can set `AFL_IGNORE_PROBLEMS_COVERAGE`.
 
   - When running in the `-M` or `-S` mode, setting `AFL_IMPORT_FIRST` causes the
     fuzzer to import test cases from other instances before doing anything else.
@@ -474,7 +483,10 @@ checks or alter some of the more exotic semantics of the tool:
     output from afl-fuzz is redirected to a file or to a pipe.
 
   - Setting `AFL_NO_STARTUP_CALIBRATION` will skip the initial calibration
-    of all starting seeds, and start fuzzing at once.
+    of all starting seeds, and start fuzzing at once. Use with care, this
+    degrades the fuzzing performance!
+
+  - Setting `AFL_NO_WARN_INSTABILITY` will suppress instability warnings.
 
   - In QEMU mode (-Q) and FRIDA mode (-O), `AFL_PATH` will be searched for
     afl-qemu-trace and afl-frida-trace.so.
@@ -572,9 +584,15 @@ checks or alter some of the more exotic semantics of the tool:
     constructors in your target, you can set `AFL_EARLY_FORKSERVER`.
     Note that this is not a compile time option but a runtime option :-)
 
-  - Set `AFL_PIZZA_MODE` to 1 to enable the April 1st stats menu, set to 0
+  - Set `AFL_PIZZA_MODE` to 1 to enable the April 1st stats menu, set to -1
     to disable although it is 1st of April.
 
+  - If you need a specific interval to update fuzzer_stats file, you can
+    set `AFL_FUZZER_STATS_UPDATE_INTERVAL` to the interval in seconds you'd
+    the file to be updated.
+    Note that will not be exact and with slow targets it can take seconds
+    until there is a slice for the time test.
+
 ## 5) Settings for afl-qemu-trace
 
 The QEMU wrapper used to instrument binary-only code supports several settings:
@@ -662,6 +680,8 @@ support.
 * `AFL_FRIDA_INST_JIT` - Enable the instrumentation of Just-In-Time compiled
   code. Code is considered to be JIT if the executable segment is not backed by
   a file.
+* `AFL_FRIDA_INST_NO_DYNAMIC_LOAD` - Don't instrument the code loaded late at
+  runtime. Strictly limits instrumentation to what has been included.
 * `AFL_FRIDA_INST_NO_OPTIMIZE` - Don't use optimized inline assembly coverage
   instrumentation (the default where available). Required to use
   `AFL_FRIDA_INST_TRACE`.
diff --git a/docs/fuzzing_binary-only_targets.md b/docs/fuzzing_binary-only_targets.md
index 266920e6..9d9d6bb6 100644
--- a/docs/fuzzing_binary-only_targets.md
+++ b/docs/fuzzing_binary-only_targets.md
@@ -201,10 +201,10 @@ afl-clang-fast's.
 ### RetroWrite
 
 RetroWrite is a static binary rewriter that can be combined with AFL++. If you
-have an x86_64 binary that still has its symbols (i.e., not stripped binary), is
-compiled with position independent code (PIC/PIE), and does not contain C++
-exceptions, then the RetroWrite solution might be for you. It decompiles to ASM
-files which can then be instrumented with afl-gcc.
+have an x86_64 or arm64 binary that does not contain C++ exceptions and - if
+x86_64 - still has it's symbols and compiled with position independent code
+(PIC/PIE), then the RetroWrite solution might be for you.
+It decompiles to ASM files which can then be instrumented with afl-gcc.
 
 Binaries that are statically instrumented for fuzzing using RetroWrite are close
 in performance to compiler-instrumented binaries and outperform the QEMU-based
diff --git a/docs/fuzzing_in_depth.md b/docs/fuzzing_in_depth.md
index a0bf1566..f75ca5dc 100644
--- a/docs/fuzzing_in_depth.md
+++ b/docs/fuzzing_in_depth.md
@@ -534,6 +534,8 @@ dictionaries/FORMAT.dict`.
 * With `afl-clang-fast`, you can set
   `AFL_LLVM_DICT2FILE=/full/path/to/new/file.dic` to automatically generate a
   dictionary during target compilation.
+  Adding `AFL_LLVM_DICT2FILE_NO_MAIN=1` to not parse main (usually command line
+  parameter parsing) is often a good idea too.
 * You also have the option to generate a dictionary yourself during an
   independent run of the target, see
   [utils/libtokencap/README.md](../utils/libtokencap/README.md).
@@ -628,7 +630,8 @@ If you have a large corpus, a corpus from a previous run or are fuzzing in a CI,
 then also set `export AFL_CMPLOG_ONLY_NEW=1` and `export AFL_FAST_CAL=1`.
 If the queue in the CI is huge and/or the execution time is slow then you can
 also add `AFL_NO_STARTUP_CALIBRATION=1` to skip the initial queue calibration
-phase and start fuzzing at once.
+phase and start fuzzing at once - but only do this if the calibration phase
+would be too long for your fuzz run time.
 
 You can also use different fuzzers. If you are using AFL spinoffs or AFL
 conforming fuzzers, then just use the same -o directory and give it a unique
@@ -900,6 +903,13 @@ then color-codes the input based on which sections appear to be critical and
 which are not; while not bulletproof, it can often offer quick insights into
 complex file formats.
 
+`casr-afl` from [CASR](https://github.com/ispras/casr) tools provides
+comfortable triaging for crashes found by AFL++. Reports are clustered and
+contain severity and other information.
+```shell
+casr-afl -i /path/to/afl/out/dir -o /path/to/casr/out/dir
+```
+
 ## 5. CI fuzzing
 
 Some notes on continuous integration (CI) fuzzing - this fuzzing is different to
@@ -907,7 +917,8 @@ normal fuzzing campaigns as these are much shorter runnings.
 
 If the queue in the CI is huge and/or the execution time is slow then you can
 also add `AFL_NO_STARTUP_CALIBRATION=1` to skip the initial queue calibration
-phase and start fuzzing at once.
+phase and start fuzzing at once. But only do that if the calibration time is
+too long for your overall available fuzz run time.
 
 1. Always:
     * LTO has a much longer compile time which is diametrical to short fuzzing -
@@ -928,7 +939,7 @@ phase and start fuzzing at once.
 3. Also randomize the afl-fuzz runtime options, e.g.:
     * 65% for `AFL_DISABLE_TRIM`
     * 50% for `AFL_KEEP_TIMEOUTS`
-    * 50% use a dictionary generated by `AFL_LLVM_DICT2FILE`
+    * 50% use a dictionary generated by `AFL_LLVM_DICT2FILE` + `AFL_LLVM_DICT2FILE_NO_MAIN=1`
     * 40% use MOpt (`-L 0`)
     * 40% for `AFL_EXPAND_HAVOC_NOW`
     * 20% for old queue processing (`-Z`)
diff --git a/docs/ideas.md b/docs/ideas.md
index b5de637f..4e419b80 100644
--- a/docs/ideas.md
+++ b/docs/ideas.md
@@ -3,6 +3,8 @@
 In the following, we describe a variety of ideas that could be implemented for
 future AFL++ versions.
 
+**NOTE:** Our GSoC participation is concerning [libafl](https://github.com/AFLplusplus/libafl), not AFL++.
+
 ## Analysis software
 
 Currently analysis is done by using afl-plot, which is rather outdated. A GTK or
@@ -16,17 +18,6 @@ and Y axis, zoom factor, log scaling on-off, etc.
 
 Mentor: vanhauser-thc
 
-## WASM Instrumentation
-
-Currently, AFL++ can be used for source code fuzzing and traditional binaries.
-With the rise of WASM as a compile target, however, a novel way of
-instrumentation needs to be implemented for binaries compiled to Webassembly.
-This can either be done by inserting instrumentation directly into the WASM AST,
-or by patching feedback into a WASM VM of choice, similar to the current Unicorn
-instrumentation.
-
-Mentor: any
-
 ## Support other programming languages
 
 Other programming languages also use llvm hence they could be (easily?)
diff --git a/docs/third_party_tools.md b/docs/third_party_tools.md
index 1175d9e5..02a40ce5 100644
--- a/docs/third_party_tools.md
+++ b/docs/third_party_tools.md
@@ -1,5 +1,10 @@
 # Tools that help fuzzing with AFL++
 
+## AFL++ and other development languages
+
+* [afl-rs](https://github.com/rust-fuzz/afl.rs) - AFL++ for RUST
+* [WASM](https://github.com/fgsect/WAFL) - AFL++ for WASM
+
 ## Speeding up fuzzing
 
 * [libfiowrapper](https://github.com/marekzmyslowski/libfiowrapper) - if the
@@ -62,3 +67,5 @@
   generates builds of debian packages suitable for AFL.
 * [afl-fid](https://github.com/FoRTE-Research/afl-fid) - a set of tools for
   working with input data.
+* [CASR](https://github.com/ispras/casr) - a set of tools for crash triage and
+  analysis.