Merge branch 'dev' of github.com:AFLplusplus/AFLplusplus into dev

9 files changed, 89 insertions, 35 deletions

diff --git a/docs/Changelog.md b/docs/Changelog.md
index a69f2ff4..5201eb8b 100644
--- a/docs/Changelog.md
+++ b/docs/Changelog.md
@@ -11,6 +11,7 @@ sending a mail to <afl-users+subscribe@googlegroups.com>.
 
 ### Version ++3.00a (develop)
   - llvm_mode/ and gcc_plugin/ moved to instrumentation/
+  - examples/ renamed to utils/
   - all compilers combined to afl-cc which emulates the previous ones
   - afl-llvm/gcc-rt.o merged into afl-compiler-rt.o
   - afl-fuzz
@@ -21,40 +22,50 @@ sending a mail to <afl-users+subscribe@googlegroups.com>.
       a schedule performance score, which is much better that the previous
       walk the whole queue approach. Select the old mode with -Z (auto enabled
       with -M)
-    - rpc.statsd support by Edznux, thanks a lot!
+    - rpc.statsd support, for stats and charts, by Edznux, thanks a lot!
     - Marcel Boehme submitted a patch that improves all AFFast schedules :)
     - not specifying -M or -S will now auto-set "-S default"
     - reading testcases from -i now descends into subdirectories
-    - allow up to 4 times the -x command line option
-    - loaded extras now have a duplicate protection
+    - allow the -x command line option up to 4 times
+    - loaded extras now have a duplication protection
     - If test cases are too large we do a partial read on the maximum
       supported size
     - longer seeds with the same trace information will now be ignored
       for fuzzing but still be used for splicing
     - crashing seeds are now not prohibiting a run anymore but are
-      skipped. They are used for splicing though.
+      skipped - they are used for splicing, though
     - update MOpt for expanded havoc modes
+    - setting the env var AFL_NO_AUTODICT will not load an LTO autodictionary
     - added NO_SPLICING compile option and makefile define
     - added INTROSPECTION make target that writes all mutations to
       out/NAME/introspection.txt
-    - added INTROSPECTION support for custom modules
     - print special compile time options used in help output
+    - when using -c cmplog, one of the childs was not killed, fixed
+    - somewhere we broke -n dumb fuzzing, fixed
+    - added afl_custom_describe to the custom mutator API to allow for easy
+      mutation reproduction on crashing inputs
   - instrumentation
     - We received an enhanced gcc_plugin module from AdaCore, thank you
       very much!!
     - not overriding -Ox or -fno-unroll-loops anymore
     - we now have our own trace-pc-guard implementation. It is the same as
       -fsanitize-coverage=trace-pc-guard from llvm 12, but: it is a) inline
-      and b) works from llvm 10+ on :)
+      and b) works from llvm 10.0.1 + onwards :)
     - new llvm pass: dict2file via AFL_LLVM_DICT2FILE, create afl-fuzz
       -x dictionary of string comparisons found during compilation
     - LTO autodict now also collects interesting cmp comparisons,
       std::string compare + find + ==, bcmp
     - fix crash in dict2file for integers > 64 bit
-  - unicornafl synced with upstream (arm64 fix, better rust bindings)
-  - added a new custom mutator: symcc -> https://github.com/eurecom-s3/symcc/
-  - added a new custom mutator: libfuzzer that integrates libfuzzer mutations
-  - Our afl++ Grammar-Mutator is now better integrated into custom_mutators/
+  - custom mutators
+    - added a new custom mutator: symcc -> https://github.com/eurecom-s3/symcc/
+    - added a new custom mutator: libfuzzer that integrates libfuzzer mutations
+    - Our afl++ Grammar-Mutator is now better integrated into custom_mutators/
+    - added INTROSPECTION support for custom modules
+    - python fuzz function was not optional, fixed
+    - some python mutator speed improvements
+  - unicornafl synced with upstream version 1.02 (fixes, better rust bindings)
+  - renamed AFL_DEBUG_CHILD_OUTPUT to AFL_DEBUG_CHILD
+  - added AFL_CRASH_EXITCODE env variable to treat a child exitcode as crash
 
 
 ### Version ++2.68c (release)
diff --git a/docs/FAQ.md b/docs/FAQ.md
index 064638f4..714d50eb 100644
--- a/docs/FAQ.md
+++ b/docs/FAQ.md
@@ -63,7 +63,7 @@ x10 - that is a x100 difference!).
 If modifying the source is not an option (e.g. because you only have a binary
 and perform binary fuzzing) you can also use a shared library with AFL_PRELOAD
 to emulate the network. This is also much faster than the real network would be.
-See [examples/socket_fuzzing/](../examples/socket_fuzzing/).
+See [utils/socket_fuzzing/](../utils/socket_fuzzing/).
 
 There is an outdated afl++ branch that implements networking if you are
 desperate though: [https://github.com/AFLplusplus/AFLplusplus/tree/networking](https://github.com/AFLplusplus/AFLplusplus/tree/networking) - 
diff --git a/docs/README.md b/docs/README.md
deleted file mode 120000
index 32d46ee8..00000000
--- a/docs/README.md
+++ /dev/null
@@ -1 +0,0 @@
-../README.md
\ No newline at end of file
diff --git a/docs/binaryonly_fuzzing.md b/docs/binaryonly_fuzzing.md
index cb1288ef..66734452 100644
--- a/docs/binaryonly_fuzzing.md
+++ b/docs/binaryonly_fuzzing.md
@@ -15,7 +15,7 @@
   high enough. Otherwise try retrowrite, afl-dyninst and if these
   fail too then try standard qemu_mode with AFL_ENTRYPOINT to where you need it.
 
-  If your target is a library use examples/afl_frida/.
+  If your target is a library use utils/afl_frida/.
 
   If your target is non-linux then use unicorn_mode/.
 
@@ -65,14 +65,14 @@
 ## AFL FRIDA
 
    If you want to fuzz a binary-only shared library then you can fuzz it with
-   frida-gum via examples/afl_frida/, you will have to write a harness to
+   frida-gum via utils/afl_frida/, you will have to write a harness to
    call the target function in the library, use afl-frida.c as a template.
 
 
 ## AFL UNTRACER
 
    If you want to fuzz a binary-only shared library then you can fuzz it with
-   examples/afl_untracer/, use afl-untracer.c as a template.
+   utils/afl_untracer/, use afl-untracer.c as a template.
    It is slower than AFL FRIDA (see above).
 
 
diff --git a/docs/custom_mutators.md b/docs/custom_mutators.md
index 2516e511..a2c544e3 100644
--- a/docs/custom_mutators.md
+++ b/docs/custom_mutators.md
@@ -34,6 +34,7 @@ C/C++:
 void *afl_custom_init(afl_state_t *afl, unsigned int seed);
 unsigned int afl_custom_fuzz_count(void *data, const unsigned char *buf, size_t buf_size);
 size_t afl_custom_fuzz(void *data, unsigned char *buf, size_t buf_size, unsigned char **out_buf, unsigned char *add_buf, size_t add_buf_size, size_t max_size);
+const char *afl_custom_describe(void *data, size_t max_description_len);
 size_t afl_custom_post_process(void *data, unsigned char *buf, size_t buf_size, unsigned char **out_buf);
 int afl_custom_init_trim(void *data, unsigned char *buf, size_t buf_size);
 size_t afl_custom_trim(void *data, unsigned char **out_buf);
@@ -57,6 +58,9 @@ def fuzz_count(buf, add_buf, max_size):
 def fuzz(buf, add_buf, max_size):
     return mutated_out
 
+def describe(max_description_length):
+    return "description_of_current_mutation"
+
 def post_process(buf):
     return out_buf
 
@@ -102,7 +106,7 @@ def introspection():
     of fuzzing attempts with this input based on a few factors.
     If however the custom mutator wants to set this number instead on how often
     it is called for a specific queue entry, use this function.
-    This function in mostly useful if **not** `AFL_CUSTOM_MUTATOR_ONLY` is used.
+    This function is most useful if `AFL_CUSTOM_MUTATOR_ONLY` is **not** used.
 
 - `fuzz` (optional):
 
@@ -110,12 +114,19 @@ def introspection():
     additional test case.
     Note that this function is optional - but it makes sense to use it.
     You would only skip this if `post_process` is used to fix checksums etc.
-    so you are using it e.g. as a post processing library.
+    so if you are using it e.g. as a post processing library.
+
+- `describe` (optional):
+
+    When this function is called, it shall describe the current testcase,
+    generated by the last mutation. This will be called, for example,
+    to name the written testcase file after a crash occurred.
+    Using it can help to reproduce crashing mutations.
 
 - `havoc_mutation` and `havoc_mutation_probability` (optional):
 
     `havoc_mutation` performs a single custom mutation on a given input. This
-    mutation is stacked with the other mutations in havoc. The other method,
+    mutation is stacked with other mutations in havoc. The other method,
     `havoc_mutation_probability`, returns the probability that `havoc_mutation`
     is called in havoc. By default, it is 6%.
 
@@ -130,6 +141,9 @@ def introspection():
     `post_process` function. This function is then transforming the data into the
     format expected by the API before executing the target.
 
+    This can return any python object that implements the buffer protocol and
+    supports PyBUF_SIMPLE. These include bytes, bytearray, etc.
+
 - `queue_new_entry` (optional):
 
     This methods is called after adding a new test case to the queue.
@@ -168,7 +182,7 @@ trimmed input. Here's a quick API description:
     on this input (e.g. if your input has n elements and you want to remove them
     one by one, return n, if you do a binary search, return log(n), and so on).
 
-    If your trimming algorithm doesn't allow you to determine the amount of
+    If your trimming algorithm doesn't allow to determine the amount of
     (remaining) steps easily (esp. while running), then you can alternatively
     return 1 here and always return 0 in `post_trim` until you are finished and
     no steps remain. In that case, returning 1 in `post_trim` will end the
@@ -210,19 +224,20 @@ Optionally, the following environment variables are supported:
 
 - `AFL_PYTHON_ONLY`
 
-    Deprecated and removed, use `AFL_CUSTOM_MUTATOR_ONLY` instead
-    trimming can cause the same test breakage like havoc and splice.
+    Deprecated and removed, use `AFL_CUSTOM_MUTATOR_ONLY` instead.
 
 - `AFL_DEBUG`
 
-    When combined with `AFL_NO_UI`, this causes the C trimming code to emit additional messages about the performance and actions of your custom trimmer. Use this to see if it works :)
+    When combined with `AFL_NO_UI`, this causes the C trimming code to emit
+    additional messages about the performance and actions of your custom
+    trimmer. Use this to see if it works :)
 
 ## 3) Usage
 
 ### Prerequisite
 
-For Python mutator, the python 3 or 2 development package is required. On
-Debian/Ubuntu/Kali this can be done:
+For Python mutators, the python 3 or 2 development package is required. On
+Debian/Ubuntu/Kali it can be installed like this:
 
 ```bash
 sudo apt install python3-dev
@@ -240,13 +255,13 @@ In case your setup is different, set the necessary variables like this:
 
 ### Custom Mutator Preparation
 
-For C/C++ mutator, the source code must be compiled as a shared object:
+For C/C++ mutators, the source code must be compiled as a shared object:
 ```bash
 gcc -shared -Wall -O3 example.c -o example.so
 ```
 Note that if you specify multiple custom mutators, the corresponding functions will
 be called in the order in which they are specified. e.g first `post_process` function of
-`example_first.so` will be called and then that of `example_second.so`
+`example_first.so` will be called and then that of `example_second.so`.
 
 ### Run
 
@@ -265,8 +280,8 @@ afl-fuzz /path/to/program
 
 ## 4) Example
 
-Please see [example.c](../examples/custom_mutators/example.c) and
-[example.py](../examples/custom_mutators/example.py)
+Please see [example.c](../utils/custom_mutators/example.c) and
+[example.py](../utils/custom_mutators/example.py)
 
 ## 5) Other Resources
 
diff --git a/docs/env_variables.md b/docs/env_variables.md
index a36f2b4e..e203055f 100644
--- a/docs/env_variables.md
+++ b/docs/env_variables.md
@@ -55,7 +55,7 @@ make fairly broad use of environmental variables instead:
     in your `$PATH`.
 
   - `AFL_PATH` can be used to point afl-gcc to an alternate location of afl-as.
-    One possible use of this is examples/clang_asm_normalize/, which lets
+    One possible use of this is utils/clang_asm_normalize/, which lets
     you instrument hand-written assembly when compiling clang code by plugging
     a normalizer into the chain. (There is no equivalent feature for GCC.)
 
@@ -294,6 +294,9 @@ checks or alter some of the more exotic semantics of the tool:
     on Linux systems. This slows things down, but lets you run more instances
     of afl-fuzz than would be prudent (if you really want to).
 
+  - Setting `AFL_NO_AUTODICT` will not load an LTO generated auto dictionary
+    that is compiled into the target.
+
   - `AFL_SKIP_CRASHES` causes AFL++ to tolerate crashing files in the input
     queue. This can help with rare situations where a program crashes only
     intermittently, but it's not really recommended under normal operating
@@ -306,6 +309,14 @@ checks or alter some of the more exotic semantics of the tool:
     don't want AFL++ to spend too much time classifying that stuff and just
     rapidly put all timeouts in that bin.
 
+  - Setting `AFL_FORKSRV_INIT_TMOUT` allows you to specify a different timeout
+    to wait for the forkserver to spin up. The default is the `-t` value times
+    `FORK_WAIT_MULT` from `config.h` (usually 10), so for a `-t 100`, the
+    default would wait for `1000` milliseconds. Setting a different time here is useful
+    if the target has a very slow startup time, for example when doing
+    full-system fuzzing or emulation, but you don't want the actual runs
+    to wait too long for timeouts.
+
   - `AFL_NO_ARITH` causes AFL++ to skip most of the deterministic arithmetics.
     This can be useful to speed up the fuzzing of text-based file formats.
 
@@ -380,8 +391,12 @@ checks or alter some of the more exotic semantics of the tool:
     processing the first queue entry; and `AFL_BENCH_UNTIL_CRASH` causes it to
     exit soon after the first crash is found.
 
-  - Setting `AFL_DEBUG_CHILD_OUTPUT` will not suppress the child output.
+  - Setting `AFL_DEBUG_CHILD` will not suppress the child output.
+    This lets you see all output of the child, making setup issues obvious.
+    For example, in an unicornafl harness, you might see python stacktraces.
+    You may also see other logs that way, indicating why the forkserver won't start.
     Not pretty but good for debugging purposes.
+    Note that `AFL_DEBUG_CHILD_OUTPUT` is deprecated.
 
   - Setting `AFL_NO_CPU_RED` will not display very high cpu usages in red color.
 
@@ -389,6 +404,13 @@ checks or alter some of the more exotic semantics of the tool:
     for an existing out folder, even if a different `-i` was provided.
     Without this setting, afl-fuzz will refuse execution for a long-fuzzed out dir.
 
+  - Setting `AFL_MAX_DET_EXRAS` will change the threshold at what number of elements
+    in the `-x` dictionary and LTO autodict (combined) the probabilistic mode will
+    kick off. In probabilistic mode not all dictionary entires will be used all
+    of the times for fuzzing mutations to not slow down fuzzing.
+    The default count is `200` elements. So for the 200 + 1st element, there is a
+    1 in 201 chance, that one of the dictionary entries will not be used directly.
+
   - Setting `AFL_NO_FORKSRV` disables the forkserver optimization, reverting to
     fork + execve() call for every tested input. This is useful mostly when
     working with unruly libraries that create threads or do other crazy
@@ -406,6 +428,13 @@ checks or alter some of the more exotic semantics of the tool:
     matches your StatsD server.
     Available flavors are `dogstatsd`, `librato`, `signalfx` and `influxdb`.
 
+  - Setting `AFL_CRASH_EXITCODE` sets the exit code afl treats as crash.
+    For example, if `AFL_CRASH_EXITCODE='-1'` is set, each input resulting
+    in an `-1` return code (i.e. `exit(-1)` got called), will be treated
+    as if a crash had ocurred.
+    This may be beneficial if you look for higher-level faulty conditions in which your
+    target still exits gracefully.
+
   - Outdated environment variables that are not supported anymore:
     `AFL_DEFER_FORKSRV`
     `AFL_PERSISTENT`
diff --git a/docs/life_pro_tips.md b/docs/life_pro_tips.md
index 323f16f1..77845c63 100644
--- a/docs/life_pro_tips.md
+++ b/docs/life_pro_tips.md
@@ -78,10 +78,10 @@ Be sure to check out docs/sister_projects.md before writing your own.
 
 ## Need to fuzz the command-line arguments of a particular program?
 
-You can find a simple solution in examples/argv_fuzzing.
+You can find a simple solution in utils/argv_fuzzing.
 
 ## Attacking a format that uses checksums? 
 
 Remove the checksum-checking code or use a postprocessor!
-See examples/custom_mutators/ for more.
+See utils/custom_mutators/ for more.
 
diff --git a/docs/notes_for_asan.md b/docs/notes_for_asan.md
index 2e18c15f..f08ae3fb 100644
--- a/docs/notes_for_asan.md
+++ b/docs/notes_for_asan.md
@@ -20,7 +20,7 @@ Because of this, fuzzing with ASAN is recommended only in four scenarios:
     - Precisely gauge memory needs using http://jwilk.net/software/recidivm .
 
     - Limit the memory available to process using cgroups on Linux (see
-      examples/asan_cgroups).
+      utils/asan_cgroups).
 
 To compile with ASAN, set AFL_USE_ASAN=1 before calling 'make clean all'. The
 afl-gcc / afl-clang wrappers will pick that up and add the appropriate flags.
@@ -74,7 +74,7 @@ There are also cgroups, but they are Linux-specific, not universally available
 even on Linux systems, and they require root permissions to set up; I'm a bit
 hesitant to make afl-fuzz require root permissions just for that. That said,
 if you are on Linux and want to use cgroups, check out the contributed script
-that ships in examples/asan_cgroups/.
+that ships in utils/asan_cgroups/.
 
 In settings where cgroups aren't available, we have no nice, portable way to
 avoid counting the ASAN allocation toward the limit. On 32-bit systems, or for
diff --git a/docs/parallel_fuzzing.md b/docs/parallel_fuzzing.md
index bf57ace8..8f2afe1b 100644
--- a/docs/parallel_fuzzing.md
+++ b/docs/parallel_fuzzing.md
@@ -152,7 +152,7 @@ write a simple script that performs two actions:
     done
     ```
 
-There is an example of such a script in examples/distributed_fuzzing/.
+There is an example of such a script in utils/distributed_fuzzing/.
 
 There are other (older) more featured, experimental tools:
   * https://github.com/richo/roving