Merge branch 'master-upstream' into custom_mutator_docs

# Conflicts:
#	afl-fuzz.c

4 files changed, 40 insertions, 116 deletions

diff --git a/docs/ChangeLog b/docs/ChangeLog
index 5e78610e..66f71a42 100644
--- a/docs/ChangeLog
+++ b/docs/ChangeLog
@@ -17,14 +17,30 @@ sending a mail to <afl-users+subscribe@googlegroups.com>.
 Version ++2.53d (dev):
 ----------------------
 
+  - big code refactoring:
+    * all includes are now in include/
+    * all afl sources are now in src/ - see src/README.src
+    * afl-fuzz was splitted up in various individual files for including
+      functionality in other programs (e.g. forkserver, memory map, etc.)
+      for better readability.
+    * new code indention everywhere
+  - auto-generating man pages for all (main) tools
+  - added AFL_FORCE_UI to show the UI even if the terminal is not detected
+  - llvm 9 is now supported (still needs testing)
+  - Android is now supported (thank to JoeyJiao!) - still need to modify the Makefile though
+  - fix building qemu on some Ubuntus (thanks to floyd!)
   - custom mutator by a loaded library is now supported (thanks to kyakdan!)
+  - added PR that includes peak_rss_mb and slowest_exec_ms in the fuzzer_stats report
+  - more support for *BSD (thanks to devnexen!)
+  - fix building on *BSD (thanks to tobias.kortkamp for the patch)
   - fix for a few features to support different map sized than 2^16
   - afl-showmap: new option -r now shows the real values in the buckets (stock
     afl never did), plus shows tuple content summary information now
-  - fix building on *BSD (thanks to tobias.kortkamp for the patch)
   - small docu updates
-  - ... your patch? :)
-
+  - NeverZero counters for QEMU
+  - NeverZero counters for Unicorn
+  - CompareCoverage Unicorn
+  - immediates-only instrumentation for CompareCoverage
 
 
 --------------------------
diff --git a/docs/env_variables.txt b/docs/env_variables.txt
index 36fdc369..cea3597b 100644
--- a/docs/env_variables.txt
+++ b/docs/env_variables.txt
@@ -223,6 +223,9 @@ checks or alter some of the more exotic semantics of the tool:
     some basic stats. This behavior is also automatically triggered when the
     output from afl-fuzz is redirected to a file or to a pipe.
 
+  - Setting AFL_FORCE_UI will force painting the UI on the screen even if
+    no valid terminal was detected (for virtual consoles)
+
   - If you are Jakub, you may need AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES.
     Others need not apply.
 
@@ -245,9 +248,19 @@ The QEMU wrapper used to instrument binary-only code supports several settings:
   - Setting AFL_INST_LIBS causes the translator to also instrument the code
     inside any dynamically linked libraries (notably including glibc).
   
+  - Setting AFL_COMPCOV_LEVEL enables the CompareCoverage tracing of all cmp
+    and sub in x86 and x86_64 and memory comparions functions (e.g. strcmp,
+    memcmp, ...) when libcompcov is preloaded using AFL_PRELOAD.
+    More info at qemu_mode/libcompcov/README.compcov.
+    There are two levels at the moment, AFL_COMPCOV_LEVEL=1 that instruments
+    only comparisons with immediate values / read-only memory and
+    AFL_COMPCOV_LEVEL=2 that instruments all the comparions. Level 2 is more
+    accurate but may need a larger shared memory.
+  
   - Setting AFL_QEMU_COMPCOV enables the CompareCoverage tracing of all
-    cmp and sub in x86 and x86_64. Support for other architectures and
-    comparison functions (mem/strcmp et al.) is planned.
+    cmp and sub in x86 and x86_64.
+    This is an alias of AFL_COMPCOV_LEVEL=1 when AFL_COMPCOV_LEVEL is
+    not specified.
 
   - The underlying QEMU binary will recognize any standard "user space
     emulation" variables (e.g., QEMU_STACK_SIZE), but there should be no
@@ -257,9 +270,10 @@ The QEMU wrapper used to instrument binary-only code supports several settings:
     Use this if you are unsure if the entrypoint might be wrong - but
     use it directly, e.g. afl-qemu-trace ./program 
 
-  - If you want to specify a specific entrypoint into the binary (this can
-    be very good for the performance!), use AFL_ENTRYPOINT for this.
+  - AFL_ENTRYPOINT allows you to specify a specific entrypoint into the
+    binary (this can be very good for the performance!).
     The entrypoint is specified as hex address, e.g. 0x4004110
+    Note that the address must be the address of a basic block.
 
 5) Settings for afl-cmin
 ------------------------
diff --git a/docs/status_screen.txt b/docs/status_screen.txt
index 37a39a99..c6f9f791 100644
--- a/docs/status_screen.txt
+++ b/docs/status_screen.txt
@@ -407,6 +407,9 @@ directory. This includes:
   - variable_paths - number of test cases showing variable behavior
   - unique_crashes - number of unique crashes recorded
   - unique_hangs   - number of unique hangs encountered
+  - command_line   - full command line used for the fuzzing session
+  - slowest_exec_ms- real time of the slowest execution in seconds
+  - peak_rss_mb    - max rss usage reached during fuzzing in MB
 
 Most of these map directly to the UI elements discussed earlier on.
 
diff --git a/docs/unicorn_mode.txt b/docs/unicorn_mode.txt
deleted file mode 100644
index b691fff8..00000000
--- a/docs/unicorn_mode.txt
+++ /dev/null
@@ -1,109 +0,0 @@
-=========================================================
-Unicorn-based binary-only instrumentation for afl-fuzz
-=========================================================
-
-1) Introduction
----------------
-
-The code in ./unicorn_mode allows you to build a standalone feature that
-leverages the Unicorn Engine and allows callers to obtain instrumentation 
-output for black-box, closed-source binary code snippets. This mechanism 
-can be then used by afl-fuzz to stress-test targets that couldn't be built 
-with afl-gcc or used in QEMU mode, or with other extensions such as 
-TriforceAFL.
-
-There is a significant performance penalty compared to native AFL,
-but at least we're able to use AFL on these binaries, right?
-
-The idea and much of the implementation comes from Nathan Voss <njvoss299@gmail.com>.
-
-2) How to use
--------------
-
-Requirements: you need an installed python2 environment.
-
-*** Building AFL's Unicorn Mode ***
-
-First, make afl as usual.
-Once that completes successfully you need to build and add in the Unicorn Mode 
-features:
-
-  $ cd unicorn_mode
-  $ ./build_unicorn_support.sh
-
-NOTE: This script downloads a recent Unicorn Engine commit that has been tested 
-and is stable-ish from the Unicorn github page. If you are offline, you'll need 
-to hack up this script a little bit and supply your own copy of Unicorn's latest 
-stable release. It's not very hard, just check out the beginning of the 
-build_unicorn_support.sh script and adjust as necessary.
-
-Building Unicorn will take a little bit (~5-10 minutes). Once it completes 
-it automatically compiles a sample application and verify that it works.
-
-*** Fuzzing with Unicorn Mode ***
-
-To really use unicorn-mode effectively you need to prepare the following:
-
-	* Relevant binary code to be fuzzed
-	* Knowledge of the memory map and good starting state
-	* Folder containing sample inputs to start fuzzing with
-		- Same ideas as any other AFL inputs
-		- Quality/speed of results will depend greatly on quality of starting 
-		  samples
-		- See AFL's guidance on how to create a sample corpus
-	* Unicorn-based test harness which:
-		- Adds memory map regions
-		- Loads binary code into memory		
-		- Emulates at least one instruction*
-			- Yeah, this is lame. See 'Gotchas' section below for more info		
-		- Loads and verifies data to fuzz from a command-line specified file
-			- AFL will provide mutated inputs by changing the file passed to 
-			  the test harness
-			- Presumably the data to be fuzzed is at a fixed buffer address
-			- If input constraints (size, invalid bytes, etc.) are known they 
-			  should be checked after the file is loaded. If a constraint 
-			  fails, just exit the test harness. AFL will treat the input as 
-			  'uninteresting' and move on.
-		- Sets up registers and memory state for beginning of test
-		- Emulates the interested code from beginning to end
-		- If a crash is detected, the test harness must 'crash' by 
-		  throwing a signal (SIGSEGV, SIGKILL, SIGABORT, etc.)
-
-Once you have all those things ready to go you just need to run afl-fuzz in
-'unicorn-mode' by passing in the '-U' flag:
-
-	$ afl-fuzz -U -m none -i /path/to/inputs -o /path/to/results -- ./test_harness @@
-
-The normal afl-fuzz command line format applies to everything here. Refer to
-AFL's main documentation for more info about how to use afl-fuzz effectively.
-
-For a much clearer vision of what all of this looks like, please refer to the
-sample provided in the 'unicorn_mode/samples' directory. There is also a blog
-post that goes over the basics at:
-
-https://medium.com/@njvoss299/afl-unicorn-fuzzing-arbitrary-binary-code-563ca28936bf
-
-The 'helper_scripts' directory also contains several helper scripts that allow you 
-to dump context from a running process, load it, and hook heap allocations. For details
-on how to use this check out the follow-up blog post to the one linked above.
-
-A example use of AFL-Unicorn mode is discussed in the Paper Unicorefuzz:
-https://www.usenix.org/conference/woot19/presentation/maier
-
-3) Gotchas, feedback, bugs
---------------------------
-
-To make sure that AFL's fork server starts up correctly the Unicorn test 
-harness script must emulate at least one instruction before loading the
-data that will be fuzzed from the input file. It doesn't matter what the
-instruction is, nor if it is valid. This is an artifact of how the fork-server
-is started and could likely be fixed with some clever re-arranging of the
-patches applied to Unicorn.
-
-Running the build script builds Unicorn and its python bindings and installs 
-them on your system. This installation will supersede any existing Unicorn
-installation with the patched afl-unicorn version.
-
-Refer to the unicorn_mode/samples/arm_example/arm_tester.c for an example
-of how to do this properly! If you don't get this right, AFL will not 
-load any mutated inputs and your fuzzing will be useless!