Merge pull request #617 from AFLplusplus/dev

push to stable

9 files changed, 64 insertions, 33 deletions

diff --git a/docs/Changelog.md b/docs/Changelog.md
index 9426ed54..4470388e 100644
--- a/docs/Changelog.md
+++ b/docs/Changelog.md
@@ -11,34 +11,41 @@ sending a mail to <afl-users+subscribe@googlegroups.com>.
 
 ### Version ++3.00a (develop)
   - llvm_mode/ and gcc_plugin/ moved to instrumentation/
+  - examples/ renamed to utils/
+  - moved libdislocator, libtokencap and qdbi_mode to utils/
   - all compilers combined to afl-cc which emulates the previous ones
   - afl-llvm/gcc-rt.o merged into afl-compiler-rt.o
   - afl-fuzz
-    - memory limits are now disabled by default, set them with -m if required
+    - not specifying -M or -S will now auto-set "-S default"
     - deterministic fuzzing is now disabled by default and can be enabled with
       -D. It is still enabled by default for -M.
     - a new seed selection was implemented that uses weighted randoms based on
       a schedule performance score, which is much better that the previous
       walk the whole queue approach. Select the old mode with -Z (auto enabled
       with -M)
-    - rpc.statsd support by Edznux, thanks a lot!
     - Marcel Boehme submitted a patch that improves all AFFast schedules :)
-    - not specifying -M or -S will now auto-set "-S default"
+    - the default schedule is now FAST
+    - memory limits are now disabled by default, set them with -m if required
+    - rpc.statsd support, for stats and charts, by Edznux, thanks a lot!
     - reading testcases from -i now descends into subdirectories
-    - allow up to 4 times the -x command line option
-    - loaded extras now have a duplicate protection
+    - allow the -x command line option up to 4 times
+    - loaded extras now have a duplication protection
     - If test cases are too large we do a partial read on the maximum
       supported size
     - longer seeds with the same trace information will now be ignored
       for fuzzing but still be used for splicing
     - crashing seeds are now not prohibiting a run anymore but are
-      skipped. They are used for splicing though.
+      skipped - they are used for splicing, though
     - update MOpt for expanded havoc modes
+    - setting the env var AFL_NO_AUTODICT will not load an LTO autodictionary
     - added NO_SPLICING compile option and makefile define
     - added INTROSPECTION make target that writes all mutations to
       out/NAME/introspection.txt
     - print special compile time options used in help output
+    - when using -c cmplog, one of the childs was not killed, fixed
     - somewhere we broke -n dumb fuzzing, fixed
+    - added afl_custom_describe to the custom mutator API to allow for easy
+      mutation reproduction on crashing inputs
   - instrumentation
     - We received an enhanced gcc_plugin module from AdaCore, thank you
       very much!!
@@ -57,8 +64,11 @@ sending a mail to <afl-users+subscribe@googlegroups.com>.
     - Our afl++ Grammar-Mutator is now better integrated into custom_mutators/
     - added INTROSPECTION support for custom modules
     - python fuzz function was not optional, fixed
-  - unicornafl synced with upstream (arm64 fix, better rust bindings)
+    - some python mutator speed improvements
+  - afl-cmin/afl-cmin.bash now search first in PATH and last in AFL_PATH
+  - unicornafl synced with upstream version 1.02 (fixes, better rust bindings)
   - renamed AFL_DEBUG_CHILD_OUTPUT to AFL_DEBUG_CHILD
+  - added AFL_CRASH_EXITCODE env variable to treat a child exitcode as crash
 
 
 ### Version ++2.68c (release)
diff --git a/docs/FAQ.md b/docs/FAQ.md
index 064638f4..714d50eb 100644
--- a/docs/FAQ.md
+++ b/docs/FAQ.md
@@ -63,7 +63,7 @@ x10 - that is a x100 difference!).
 If modifying the source is not an option (e.g. because you only have a binary
 and perform binary fuzzing) you can also use a shared library with AFL_PRELOAD
 to emulate the network. This is also much faster than the real network would be.
-See [examples/socket_fuzzing/](../examples/socket_fuzzing/).
+See [utils/socket_fuzzing/](../utils/socket_fuzzing/).
 
 There is an outdated afl++ branch that implements networking if you are
 desperate though: [https://github.com/AFLplusplus/AFLplusplus/tree/networking](https://github.com/AFLplusplus/AFLplusplus/tree/networking) - 
diff --git a/docs/README.md b/docs/README.md
deleted file mode 120000
index 32d46ee8..00000000
--- a/docs/README.md
+++ /dev/null
@@ -1 +0,0 @@
-../README.md
\ No newline at end of file
diff --git a/docs/binaryonly_fuzzing.md b/docs/binaryonly_fuzzing.md
index cb1288ef..66734452 100644
--- a/docs/binaryonly_fuzzing.md
+++ b/docs/binaryonly_fuzzing.md
@@ -15,7 +15,7 @@
   high enough. Otherwise try retrowrite, afl-dyninst and if these
   fail too then try standard qemu_mode with AFL_ENTRYPOINT to where you need it.
 
-  If your target is a library use examples/afl_frida/.
+  If your target is a library use utils/afl_frida/.
 
   If your target is non-linux then use unicorn_mode/.
 
@@ -65,14 +65,14 @@
 ## AFL FRIDA
 
    If you want to fuzz a binary-only shared library then you can fuzz it with
-   frida-gum via examples/afl_frida/, you will have to write a harness to
+   frida-gum via utils/afl_frida/, you will have to write a harness to
    call the target function in the library, use afl-frida.c as a template.
 
 
 ## AFL UNTRACER
 
    If you want to fuzz a binary-only shared library then you can fuzz it with
-   examples/afl_untracer/, use afl-untracer.c as a template.
+   utils/afl_untracer/, use afl-untracer.c as a template.
    It is slower than AFL FRIDA (see above).
 
 
diff --git a/docs/custom_mutators.md b/docs/custom_mutators.md
index 53f783fe..a2c544e3 100644
--- a/docs/custom_mutators.md
+++ b/docs/custom_mutators.md
@@ -34,6 +34,7 @@ C/C++:
 void *afl_custom_init(afl_state_t *afl, unsigned int seed);
 unsigned int afl_custom_fuzz_count(void *data, const unsigned char *buf, size_t buf_size);
 size_t afl_custom_fuzz(void *data, unsigned char *buf, size_t buf_size, unsigned char **out_buf, unsigned char *add_buf, size_t add_buf_size, size_t max_size);
+const char *afl_custom_describe(void *data, size_t max_description_len);
 size_t afl_custom_post_process(void *data, unsigned char *buf, size_t buf_size, unsigned char **out_buf);
 int afl_custom_init_trim(void *data, unsigned char *buf, size_t buf_size);
 size_t afl_custom_trim(void *data, unsigned char **out_buf);
@@ -57,6 +58,9 @@ def fuzz_count(buf, add_buf, max_size):
 def fuzz(buf, add_buf, max_size):
     return mutated_out
 
+def describe(max_description_length):
+    return "description_of_current_mutation"
+
 def post_process(buf):
     return out_buf
 
@@ -102,7 +106,7 @@ def introspection():
     of fuzzing attempts with this input based on a few factors.
     If however the custom mutator wants to set this number instead on how often
     it is called for a specific queue entry, use this function.
-    This function in mostly useful if **not** `AFL_CUSTOM_MUTATOR_ONLY` is used.
+    This function is most useful if `AFL_CUSTOM_MUTATOR_ONLY` is **not** used.
 
 - `fuzz` (optional):
 
@@ -110,12 +114,19 @@ def introspection():
     additional test case.
     Note that this function is optional - but it makes sense to use it.
     You would only skip this if `post_process` is used to fix checksums etc.
-    so you are using it e.g. as a post processing library.
+    so if you are using it e.g. as a post processing library.
+
+- `describe` (optional):
+
+    When this function is called, it shall describe the current testcase,
+    generated by the last mutation. This will be called, for example,
+    to name the written testcase file after a crash occurred.
+    Using it can help to reproduce crashing mutations.
 
 - `havoc_mutation` and `havoc_mutation_probability` (optional):
 
     `havoc_mutation` performs a single custom mutation on a given input. This
-    mutation is stacked with the other mutations in havoc. The other method,
+    mutation is stacked with other mutations in havoc. The other method,
     `havoc_mutation_probability`, returns the probability that `havoc_mutation`
     is called in havoc. By default, it is 6%.
 
@@ -171,7 +182,7 @@ trimmed input. Here's a quick API description:
     on this input (e.g. if your input has n elements and you want to remove them
     one by one, return n, if you do a binary search, return log(n), and so on).
 
-    If your trimming algorithm doesn't allow you to determine the amount of
+    If your trimming algorithm doesn't allow to determine the amount of
     (remaining) steps easily (esp. while running), then you can alternatively
     return 1 here and always return 0 in `post_trim` until you are finished and
     no steps remain. In that case, returning 1 in `post_trim` will end the
@@ -213,19 +224,20 @@ Optionally, the following environment variables are supported:
 
 - `AFL_PYTHON_ONLY`
 
-    Deprecated and removed, use `AFL_CUSTOM_MUTATOR_ONLY` instead
-    trimming can cause the same test breakage like havoc and splice.
+    Deprecated and removed, use `AFL_CUSTOM_MUTATOR_ONLY` instead.
 
 - `AFL_DEBUG`
 
-    When combined with `AFL_NO_UI`, this causes the C trimming code to emit additional messages about the performance and actions of your custom trimmer. Use this to see if it works :)
+    When combined with `AFL_NO_UI`, this causes the C trimming code to emit
+    additional messages about the performance and actions of your custom
+    trimmer. Use this to see if it works :)
 
 ## 3) Usage
 
 ### Prerequisite
 
-For Python mutator, the python 3 or 2 development package is required. On
-Debian/Ubuntu/Kali this can be done:
+For Python mutators, the python 3 or 2 development package is required. On
+Debian/Ubuntu/Kali it can be installed like this:
 
 ```bash
 sudo apt install python3-dev
@@ -243,13 +255,13 @@ In case your setup is different, set the necessary variables like this:
 
 ### Custom Mutator Preparation
 
-For C/C++ mutator, the source code must be compiled as a shared object:
+For C/C++ mutators, the source code must be compiled as a shared object:
 ```bash
 gcc -shared -Wall -O3 example.c -o example.so
 ```
 Note that if you specify multiple custom mutators, the corresponding functions will
 be called in the order in which they are specified. e.g first `post_process` function of
-`example_first.so` will be called and then that of `example_second.so`
+`example_first.so` will be called and then that of `example_second.so`.
 
 ### Run
 
@@ -268,8 +280,8 @@ afl-fuzz /path/to/program
 
 ## 4) Example
 
-Please see [example.c](../examples/custom_mutators/example.c) and
-[example.py](../examples/custom_mutators/example.py)
+Please see [example.c](../utils/custom_mutators/example.c) and
+[example.py](../utils/custom_mutators/example.py)
 
 ## 5) Other Resources
 
diff --git a/docs/env_variables.md b/docs/env_variables.md
index 04ba032a..e203055f 100644
--- a/docs/env_variables.md
+++ b/docs/env_variables.md
@@ -55,7 +55,7 @@ make fairly broad use of environmental variables instead:
     in your `$PATH`.
 
   - `AFL_PATH` can be used to point afl-gcc to an alternate location of afl-as.
-    One possible use of this is examples/clang_asm_normalize/, which lets
+    One possible use of this is utils/clang_asm_normalize/, which lets
     you instrument hand-written assembly when compiling clang code by plugging
     a normalizer into the chain. (There is no equivalent feature for GCC.)
 
@@ -294,6 +294,9 @@ checks or alter some of the more exotic semantics of the tool:
     on Linux systems. This slows things down, but lets you run more instances
     of afl-fuzz than would be prudent (if you really want to).
 
+  - Setting `AFL_NO_AUTODICT` will not load an LTO generated auto dictionary
+    that is compiled into the target.
+
   - `AFL_SKIP_CRASHES` causes AFL++ to tolerate crashing files in the input
     queue. This can help with rare situations where a program crashes only
     intermittently, but it's not really recommended under normal operating
@@ -425,6 +428,13 @@ checks or alter some of the more exotic semantics of the tool:
     matches your StatsD server.
     Available flavors are `dogstatsd`, `librato`, `signalfx` and `influxdb`.
 
+  - Setting `AFL_CRASH_EXITCODE` sets the exit code afl treats as crash.
+    For example, if `AFL_CRASH_EXITCODE='-1'` is set, each input resulting
+    in an `-1` return code (i.e. `exit(-1)` got called), will be treated
+    as if a crash had ocurred.
+    This may be beneficial if you look for higher-level faulty conditions in which your
+    target still exits gracefully.
+
   - Outdated environment variables that are not supported anymore:
     `AFL_DEFER_FORKSRV`
     `AFL_PERSISTENT`
diff --git a/docs/life_pro_tips.md b/docs/life_pro_tips.md
index 323f16f1..50ad75d4 100644
--- a/docs/life_pro_tips.md
+++ b/docs/life_pro_tips.md
@@ -13,7 +13,7 @@ See [parallel_fuzzing.md](parallel_fuzzing.md) for step-by-step tips.
 
 ## Improve the odds of spotting memory corruption bugs with libdislocator.so!
 
-It's easy. Consult [libdislocator/README.md](../libdislocator/README.md) for usage tips.
+It's easy. Consult [utils/libdislocator/README.md](../utils/libdislocator/README.md) for usage tips.
 
 ## Want to understand how your target parses a particular input file?
 
@@ -78,10 +78,10 @@ Be sure to check out docs/sister_projects.md before writing your own.
 
 ## Need to fuzz the command-line arguments of a particular program?
 
-You can find a simple solution in examples/argv_fuzzing.
+You can find a simple solution in utils/argv_fuzzing.
 
 ## Attacking a format that uses checksums? 
 
 Remove the checksum-checking code or use a postprocessor!
-See examples/custom_mutators/ for more.
+See utils/custom_mutators/ for more.
 
diff --git a/docs/notes_for_asan.md b/docs/notes_for_asan.md
index 2e18c15f..2b3bc028 100644
--- a/docs/notes_for_asan.md
+++ b/docs/notes_for_asan.md
@@ -20,7 +20,7 @@ Because of this, fuzzing with ASAN is recommended only in four scenarios:
     - Precisely gauge memory needs using http://jwilk.net/software/recidivm .
 
     - Limit the memory available to process using cgroups on Linux (see
-      examples/asan_cgroups).
+      utils/asan_cgroups).
 
 To compile with ASAN, set AFL_USE_ASAN=1 before calling 'make clean all'. The
 afl-gcc / afl-clang wrappers will pick that up and add the appropriate flags.
@@ -35,7 +35,7 @@ no sanitizers compiled in.
 There is also the option of generating a corpus using a non-ASAN binary, and
 then feeding it to an ASAN-instrumented one to check for bugs. This is faster,
 and can give you somewhat comparable results. You can also try using
-libdislocator (see libdislocator/README.dislocator.md in the parent directory) as a
+libdislocator (see [utils/libdislocator/README.dislocator.md](../utils/libdislocator/README.dislocator.md) in the parent directory) as a
 lightweight and hassle-free (but less thorough) alternative.
 
 ## 2) Long version
@@ -74,7 +74,7 @@ There are also cgroups, but they are Linux-specific, not universally available
 even on Linux systems, and they require root permissions to set up; I'm a bit
 hesitant to make afl-fuzz require root permissions just for that. That said,
 if you are on Linux and want to use cgroups, check out the contributed script
-that ships in examples/asan_cgroups/.
+that ships in utils/asan_cgroups/.
 
 In settings where cgroups aren't available, we have no nice, portable way to
 avoid counting the ASAN allocation toward the limit. On 32-bit systems, or for
diff --git a/docs/parallel_fuzzing.md b/docs/parallel_fuzzing.md
index bf57ace8..8f2afe1b 100644
--- a/docs/parallel_fuzzing.md
+++ b/docs/parallel_fuzzing.md
@@ -152,7 +152,7 @@ write a simple script that performs two actions:
     done
     ```
 
-There is an example of such a script in examples/distributed_fuzzing/.
+There is an example of such a script in utils/distributed_fuzzing/.
 
 There are other (older) more featured, experimental tools:
   * https://github.com/richo/roving