Merge pull request #843 from AFLplusplus/tmp

Tmp

6 files changed, 293 insertions, 27 deletions

diff --git a/docs/Changelog.md b/docs/Changelog.md
index e9efdf38..6b7ebf15 100644
--- a/docs/Changelog.md
+++ b/docs/Changelog.md
@@ -8,59 +8,130 @@
 Want to stay in the loop on major new features? Join our mailing list by
 sending a mail to <afl-users+subscribe@googlegroups.com>.
 
+### Version ++3.13a (development)
+  - ...
 
-### Version ++3.01a (dev)
+### Version ++3.12c (release)
+  - afl-fuzz:
+    - added AFL_TARGET_ENV variable to pass extra env vars to the target
+      (for things like LD_LIBRARY_PATH)
+    - fix map detection, AFL_MAP_SIZE not needed anymore for most cases
+    - fix counting favorites (just a display thing)
+  - afl-cc:
+    - fix cmplog rtn (rare crash and not being able to gather ptr data)
+    - fix our own PCGUARD implementation to compile with llvm 10.0.1
+    - link runtime not to shared libs
+    - ensure shared libraries are properly built and instrumented
+    - AFL_LLVM_INSTRUMENT_ALLOW/DENY were not implemented for LTO, added
+    - show correct LLVM PCGUARD NATIVE mode when auto switching to it
+      and keep fsanitize-coverage-*list=...
+      Short mnemnonic NATIVE is now also accepted.
+  - qemu_mode (thanks @realmadsci):
+    - move AFL_PRELOAD and AFL_USE_QASAN logic inside afl-qemu-trace
+    - add AFL_QEMU_CUSTOM_BIN
+  - unicorn_mode
+    - accidently removed the subfolder from github, re-added
+  - added DEFAULT_PERMISSION to config.h for all files created, default
+    to 0600
+
+### Version ++3.11c (release)
+  - afl-fuzz:
+    - better auto detection of map size
+    - fix sanitizer settings (bug since 3.10c)
+    - fix an off-by-one overwrite in cmplog
+    - add non-unicode variants from unicode-looking dictionary entries
+    - Rust custom mutator API improvements
+    - Imported crash stats painted yellow on resume (only new ones are red)
+  - afl-cc:
+    - added AFL_NOOPT that will just pass everything to the normal
+      gcc/clang compiler without any changes - to pass weird configure
+      scripts
+    - fixed a crash that can occur with ASAN + CMPLOG together plus
+      better support for unicode (thanks to @stbergmann for reporting!)
+    - fixed a crash in LAF transform for empty strings
+    - handle erroneous setups in which multiple afl-compiler-rt are
+      compiled into the target. This now also supports dlopen()
+      instrumented libs loaded before the forkserver and even after the
+      forkserver is started (then with collisions though)
+    - the compiler rt was added also in object building (-c) which
+      should have been fixed years ago but somewhere got lost :(
+    - Renamed CTX to CALLER, added correct/real CTX implementation to
+      CLASSIC
+  - qemu_mode:
+    - added AFL_QEMU_EXCLUDE_RANGES env by @realmadsci, thanks!
+    - if no new/updated checkout is wanted, build with:
+      NO_CHECKOUT=1 ./build_qemu_support.sh
+    - we no longer perform a "git drop"
+  - afl-cmin: support filenames with spaces
+
+### Version ++3.10c (release)
   - Mac OS ARM64 support
   - Android support fixed and updated by Joey Jiaojg - thanks!
   - New selective instrumentation option with __AFL_COVERAGE_* commands
     to be placed in the source code.
     Check out instrumentation/README.instrument_list.md
   - afl-fuzz
-    - Making AFL_MAP_SIZE obsolete - afl-fuzz now learns on start the
-      target map size
+    - Making AFL_MAP_SIZE (mostly) obsolete - afl-fuzz now learns on
+      start the target map size
     - upgraded cmplog/redqueen: solving for floating point, solving
       transformations (e.g. toupper, tolower, to/from hex, xor,
-      arithmetics, etc.). this is costly hence new command line option
-      -l that sets the intensity (values 1 to 3). recommended is 1 or 2.
-    - added `AFL_CMPLOG_ONLY_NEW` to not use cmplog on initial testcases from
-      `-i` or resumes (as these have most likely already been done)
+      arithmetics, etc.). This is costly hence new command line option
+      `-l` that sets the intensity (values 1 to 3). Recommended is 2.
+    - added `AFL_CMPLOG_ONLY_NEW` to not use cmplog on initial seeds
+      from `-i` or resumes (these have most likely already been done)
     - fix crash for very, very fast targets+systems (thanks to mhlakhani
       for reporting)
-    - if determinstic mode is active (-D, or -M without -d) then we sync
-      after every queue entry as this can take very long time otherwise
+    - on restarts (`-i`)/autoresume (AFL_AUTORESUME) the stats are now
+      reloaded and used, thanks to Vimal Joseph for this patch! 
+    - changed the meaning of '+' of the '-t' option, it now means to
+      auto-calculate the timeout with the value given being the max
+      timeout. The original meaning of skipping timeouts instead of
+      abort is now inherent to the -t option.
+    - if deterministic mode is active (`-D`, or `-M` without `-d`) then
+      we sync after every queue entry as this can take very long time
+      otherwise
+    - added minimum SYNC_TIME to include/config.h (30 minutes default)
     - better detection if a target needs a large shared map
-    - fix for -Z
+    - fix for `-Z`
+    - fixed a few crashes
     - switched to an even faster RNG
     - added hghwng's patch for faster trace map analysis
+    - printing suggestions for mistyped `AFL_` env variables
+    - added Rust bindings for custom mutators (thanks @julihoh)
   - afl-cc
     - allow instrumenting LLVMFuzzerTestOneInput
     - fixed endless loop for allow/blocklist lines starting with a
       comment (thanks to Zherya for reporting)
     - cmplog/redqueen now also tracks floating point, _ExtInt() + 128bit
     - cmplog/redqueen can now process basic libc++ and libstdc++
-      std::string comparisons (though no position or length type variants)
-    - added support for __afl_coverage_interesting() for LTO and
-      and our own PCGUARD (llvm 10.0.1+), read more about this function
-      and selective coverage in instrumentation/README.instrument_list.md
+      std::string comparisons (no position or length type variants)
+    - added support for __afl_coverage_interesting() for LTO and our
+      own PCGUARD (llvm 10.0.1+), read more about this function and
+      selective coverage in instrumentation/README.instrument_list.md
     - added AFL_LLVM_INSTRUMENT option NATIVE for native clang pc-guard
       support (less performant than our own), GCC for old afl-gcc and
       CLANG for old afl-clang
+    - fixed a potential crash in the LAF feature
+    - workaround for llvm bitcast lto bug
+    - workaround for llvm 13
   - qemuafl
-    - ported QASan to qemuafl! see qemu_mode/libqasan/README.md
+    - QASan (address sanitizer for Qemu) ported to qemuafl!
+      See qemu_mode/libqasan/README.md
     - solved some persistent mode bugs (thanks Dil4rd)
     - solved an issue when dumping the memory maps (thanks wizche)
     - Android support for QASan
   - unicornafl
-    - Substential speed gains in python bindings for certain use cases
+    - Substantial speed gains in python bindings for certain use cases
     - Improved rust bindings
-    - Added a new example harness to compare python, c, and rust bindings
+    - Added a new example harness to compare python, c and rust bindings
+  - afl-cmin and afl-showmap now support the -f option
+  - afl_plot now also generates a graph on the discovered edges
   - changed default: no memory limit for afl-cmin and afl-cmin.bash
-  - warn on any _AFL and __AFL env vars
-  - LLVM mode is now compiled with -j4, unicorn with all cores. qemu was
-    already building with all cores, the gcc plugin needs only one.
+  - warn on any _AFL and __AFL env vars.
+  - set AFL_IGNORE_UNKNOWN_ENVS to not warn on unknown AFL_... env vars
   - added dummy Makefile to instrumentation/
   - Updated utils/afl_frida to be 5% faster, 7% on x86_x64
-  - Added AFL_KILL_SIGNAL env variable (thanks @v-p-b)
+  - Added `AFL_KILL_SIGNAL` env variable (thanks @v-p-b)
   - @Edznux added a nice documentation on how to use rpc.statsd with
     afl++ in docs/rpc_statsd.md, thanks!
 
diff --git a/docs/custom_mutators.md b/docs/custom_mutators.md
index a2c544e3..61d711e4 100644
--- a/docs/custom_mutators.md
+++ b/docs/custom_mutators.md
@@ -4,6 +4,11 @@ This file describes how you can implement custom mutations to be used in AFL.
 For now, we support C/C++ library and Python module, collectivelly named as the
 custom mutator.
 
+There is also experimental support for Rust in `custom_mutators/rust`.
+Please refer to that directory for documentation.
+Run ```cargo doc -p custom_mutator --open``` in that directory to view the
+documentation in your web browser.
+
 Implemented by
 - C/C++ library (`*.so`): Khaled Yakdan from Code Intelligence (<yakdan@code-intelligence.de>)
 - Python module: Christian Holler from Mozilla (<choller@mozilla.com>)
diff --git a/docs/docs.md b/docs/docs.md
new file mode 100644
index 00000000..ed6ec85e
--- /dev/null
+++ b/docs/docs.md
@@ -0,0 +1,122 @@
+# Restructure afl++'s documentation
+
+## About us
+
+We are dedicated to everything around fuzzing, our main and most well known
+contribution is the fuzzer `afl++` which is part of all major Unix
+distributions (e.g. Debian, Arch, FreeBSD, etc.) and is deployed on Google's
+oss-fuzz and clusterfuzz. It is rated the top fuzzer on Google's fuzzbench.
+
+We are four individuals from Europe supported by a large community.
+
+All our tools are open source.
+
+## About the afl++ fuzzer project
+
+afl++ inherited it's documentation from the original Google afl project.
+Since then it has been massively improved - feature and performance wise -
+and although the documenation has likewise been continued it has grown out
+of proportion.
+The documentation is done by non-natives to the English language, plus
+none of us has a writer background.
+
+We see questions on afl++ usage on mailing lists (e.g. afl-users), discord
+channels, web forums and as issues in our repository.
+
+This only increases as afl++ has been on the top of Google's fuzzbench
+statistics (which measures the performance of fuzzers) and is now being
+integrated in Google's oss-fuzz and clusterfuzz - and is in many Unix
+packaging repositories, e.g. Debian, FreeBSD, etc.
+
+afl++ now has 44 (!) documentation files with 13k total lines of content.
+This is way too much.
+
+Hence afl++ needs a complete overhaul of it's documentation, both on a 
+organisation/structural level as well as the content.
+
+Overall the following actions have to be performed:
+  * Create a better structure of documentation so it is easier to find the
+    information that is being looked for, combining and/or splitting up the
+    existing documents as needed.
+  * Rewrite some documentation to remove duplication. Several information is
+    present several times in the documentation. These should be removed to
+    where needed so that we have as little bloat as possible.
+  * The documents have been written and modified by a lot of different people,
+    most of them non-native English speaker. Hence an overall review where
+    parts should be rewritten has to be performed and then the rewrite done.
+  * Create a cheat-sheet for a very short best-setup build and run of afl++
+  * Pictures explain more than 1000 words. We need at least 4 images that
+    explain the workflow with afl++:
+      - the build workflow
+      - the fuzzing workflow
+      - the fuzzing campaign management workflow
+      - the overall workflow that is an overview of the above
+      - maybe more? where the technical writes seems it necessary for
+        understanding.
+
+Requirements:
+  * Documentation has to be in Markdown format
+  * Images have to be either in SVG or PNG format.
+  * All documentation should be (moved) in(to) docs/
+
+The project does not require writing new documentation or tutorials beside the
+cheat sheet. The technical information for the cheat sheet will be provided by
+us.
+
+## Metrics
+
+afl++ is a the highest performant fuzzer publicly available - but is also the
+most feature rich and complex. With the publicity of afl++' success and
+deployment in Google projects internally and externally and availability as
+a package on most Linux distributions we see more and more issues being
+created and help requests on our Discord channel that would not be
+necessary if people would have read through all our documentation - which
+is unrealistic.
+
+We expect the the new documenation after this project to be cleaner, easier
+accessible and lighter to digest by our users, resulting in much less
+help requests. On the other hand the amount of users using afl++ should
+increase as well as it will be more accessible which would also increase
+questions again - but overall resulting in a reduction of help requests.
+
+In numbers: we currently have per week on average 5 issues on Github,
+10 questions on discord and 1 on mailing lists that would not be necessary
+with perfect documentation and perfect people.
+
+We would consider this project a success if afterwards we only have
+2 issues on Github and 3 questions on discord anymore that would be answered
+by reading the documentation. The mailing list is usually used by the most
+novice users and we don't expect any less questions there.
+
+## Project Budget
+
+We have zero experience with technical writers, so this is very hard for us
+to calculate. We expect it to be a lot of work though because of the amount
+of documentation we have that needs to be restructured and partially rewritten
+(44 documents with 13k total lines of content).
+
+We assume the daily rate of a very good and experienced technical writer in
+times of a pandemic to be ~500$ (according to web research), and calculate
+the overall amout of work to be around 20 days for everything incl. the
+graphics (but again - this is basically just guessing).
+
+Technical Writer                                              10000$
+Volunteer stipends                                                0$ (waved)
+T-Shirts for the top 10 contributors and helpers to this documentation project:
+	10 afl++ logo t-shirts 		20$ each		200$
+	10 shipping cost of t-shirts    10$ each		100$
+
+Total: 10.300$
+(in the submission form 10.280$ was entered)
+
+## Additional Information
+
+We have participated in Google Summer of Code in 2020 and hope to be selected
+again in 2021.
+
+We have no experience with a technical writer, but we will support that person
+with video calls, chats, emails and messaging, provide all necessary information
+and write technical contents that is required for the success of this project.
+It is clear to us that a technical writer knows how to write, but cannot know
+the technical details in a complex tooling like in afl++. This guidance, input,
+etc. has to come from us.
diff --git a/docs/env_variables.md b/docs/env_variables.md
index 41733f1b..de6b4bd8 100644
--- a/docs/env_variables.md
+++ b/docs/env_variables.md
@@ -5,6 +5,10 @@
   users or for some types of custom fuzzing setups. See [README.md](README.md) for the general
   instruction manual.
 
+  Note that most tools will warn on any unknown AFL environment variables.
+  This is for warning on typos that can happen. If you want to disable this
+  check then set the `AFL_IGNORE_UNKNOWN_ENVS` environment variable.
+
 ## 1) Settings for all compilers
 
 Starting with afl++ 3.0 there is only one compiler: afl-cc
@@ -18,11 +22,21 @@ To select the different instrumentation modes this can be done by
 `MODE` can be one of `LTO` (afl-clang-lto*), `LLVM` (afl-clang-fast*), `GCC_PLUGIN`
 (afl-g*-fast) or `GCC` (afl-gcc/afl-g++).
 
-
 Because (with the exception of the --afl-MODE command line option) the
 compile-time tools do not accept afl specific command-line options, they
 make fairly broad use of environmental variables instead:
 
+  - Some build/configure scripts break with afl++ compilers. To be able to
+    pass them, do:
+```
+       export CC=afl-cc
+       export CXX=afl-c++
+       export AFL_NOOPT=1
+       ./configure --disable-shared --disabler-werror
+       unset AFL_NOOPT
+       make
+```
+
   - Most afl tools do not print any output if stdout/stderr are redirected.
     If you want to get the output into a file then set the `AFL_DEBUG`
     environment variable.
@@ -379,6 +393,10 @@ checks or alter some of the more exotic semantics of the tool:
 
   - In QEMU mode (-Q), `AFL_PATH` will be searched for afl-qemu-trace.
 
+  - In QEMU mode (-Q), setting `AFL_QEMU_CUSTOM_BIN` cause afl-fuzz to skip
+    prepending `afl-qemu-trace` to your command line. Use this if you wish to use a
+    custom afl-qemu-trace or if you need to modify the afl-qemu-trace arguments.
+
   - Setting `AFL_CYCLE_SCHEDULES` will switch to a different schedule everytime
     a cycle is finished.
 
@@ -390,6 +408,12 @@ checks or alter some of the more exotic semantics of the tool:
     without disrupting the afl-fuzz process itself. This is useful, among other
     things, for bootstrapping libdislocator.so.
 
+  - Setting `AFL_TARGET_ENV` causes AFL++ to set extra environment variables
+    for the target binary. Example: `AFL_TARGET_ENV="VAR1=1 VAR2='a b c'" afl-fuzz ... `
+    This exists mostly for things like `LD_LIBRARY_PATH` but it would theoretically
+    allow fuzzing of AFL++ itself (with 'target' AFL++ using some AFL_ vars that
+    would disrupt work of 'fuzzer' AFL++).
+
   - Setting `AFL_NO_UI` inhibits the UI altogether, and just periodically prints
     some basic stats. This behavior is also automatically triggered when the
     output from afl-fuzz is redirected to a file or to a pipe.
@@ -410,7 +434,8 @@ checks or alter some of the more exotic semantics of the tool:
     and RECORD:000000,cnt:000009 being the crash case.
 
   - If you are Jakub, you may need `AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES`.
-    Others need not apply.
+    Others need not apply, unless they also want to disable the
+    `/proc/sys/kernel/core_pattern` check.
 
   - Benchmarking only: `AFL_BENCH_JUST_ONE` causes the fuzzer to exit after
     processing the first queue entry; and `AFL_BENCH_UNTIL_CRASH` causes it to
@@ -457,6 +482,7 @@ checks or alter some of the more exotic semantics of the tool:
     `banner` corresponds to the name of the fuzzer provided through `-M/-S`.
     `afl_version` corresponds to the currently running afl version (e.g `++3.0c`).
     Default (empty/non present) will add no tags to the metrics.
+    See [rpc_statsd.md](rpc_statsd.md) for more information.
 
   - Setting `AFL_CRASH_EXITCODE` sets the exit code afl treats as crash.
     For example, if `AFL_CRASH_EXITCODE='-1'` is set, each input resulting
@@ -523,6 +549,12 @@ The QEMU wrapper used to instrument binary-only code supports several settings:
     stack pointer in which QEMU can find the return address when `start addr` is
     hit.
 
+  - With `AFL_USE_QASAN` you can enable QEMU AddressSanitizer for dynamically
+    linked binaries.
+
+  - With `AFL_QEMU_FORCE_DFL` you force QEMU to ignore the registered signal
+    handlers of the target.
+
 ## 6) Settings for afl-cmin
 
 The corpus minimization script offers very little customization:
diff --git a/docs/ideas.md b/docs/ideas.md
index 7cbe60a5..0130cf61 100644
--- a/docs/ideas.md
+++ b/docs/ideas.md
@@ -3,6 +3,42 @@
 In the following, we describe a variety of ideas that could be implemented
 for future AFL++ versions.
 
+# GSoC 2021
+
+All GSoC 2021 projects will be in the Rust development language!
+
+## UI for libaflrs
+
+Write a user interface to libaflrs, the upcoming backend of afl++.
+This might look like the afl-fuzz UI, but you can improve on it - and should!
+
+## Schedulers for libaflrs
+
+Schedulers is a mechanism that selects items from the fuzzing corpus based
+on strategy and randomness. One scheduler might focus on long paths,
+another on rarity of edges disocvered, still another on a combination on
+things. Some of the schedulers in afl++ have to be ported, but you are free
+to come up with your own if you want to - and see how it performs.
+
+## Forkserver support for libaflrs
+
+The current libaflrs implementation fuzzes in-memory, however obviously we
+want to support afl instrumented binaries as well.
+Hence a forkserver support needs to be implemented - forking off the target
+and talking to the target via a socketpair and the communication protocol
+within.
+
+## More Observers for libaflrs
+
+An observer is measuring functionality that looks at the target being fuzzed
+and documents something about it. In traditional fuzzing this is the coverage
+in the target, however we want to add various more observers, e.g. stack depth,
+heap usage, etc. - this is a topic for an experienced Rust developer.
+
+# Generic ideas and wishlist - NOT PART OF GSoC 2021 !
+
+The below list is not part of GSoC 2021.
+
 ## Analysis software
 
 Currently analysis is done by using afl-plot, which is rather outdated.
diff --git a/docs/rpc_statsd.md b/docs/rpc_statsd.md
index 02f72be6..fb97aa09 100644
--- a/docs/rpc_statsd.md
+++ b/docs/rpc_statsd.md
@@ -1,6 +1,6 @@
 # Remote monitoring with StatsD
 
-StatsD allows you to receive and aggregate metrics from a wide range of application and retransmit them to the backend of your choice.
+StatsD allows you to receive and aggregate metrics from a wide range of applications and retransmit them to the backend of your choice.
 This enables you to create nice and readable dashboards containing all the information you need on your fuzzer instances.
 No need to write your own statistics parsing system, deploy and maintain it to all your instances, sync with your graph rendering system...
 
@@ -45,7 +45,7 @@ For more information on these env vars, check out `docs/env_variables.md`.
 
 The simplest way of using this feature is to use any metric provider and change the host/port of your StatsD daemon,
 with `AFL_STATSD_HOST` and `AFL_STATSD_PORT`, if required (defaults are `localhost` and port `8125`).
-To get started, here are some instruction with free and open source tools.
+To get started, here are some instructions with free and open source tools.
 The following setup is based on Prometheus, statsd_exporter and Grafana.
 Grafana here is not mandatory, but gives you some nice graphs and features.
 
@@ -131,7 +131,7 @@ mappings:
 
 Run `docker-compose up -d`.
 
-Everything should be now setup, you are now able to run your fuzzers with
+Everything should now be setup, you are now able to run your fuzzers with
 
 ```
 AFL_STATSD_TAGS_FLAVOR=dogstatsd AFL_STATSD=1 afl-fuzz -M test-fuzzer-1 -i i -o o ./bin/my-application @@
@@ -139,5 +139,5 @@ AFL_STATSD_TAGS_FLAVOR=dogstatsd AFL_STATSD=1 afl-fuzz -S test-fuzzer-2 -i i -o
 ...
 ```
 
-This setup may be modified before use in production environment. Depending on your needs: addind passwords, creating volumes for storage,
+This setup may be modified before use in a production environment. Depending on your needs: adding passwords, creating volumes for storage,
 tweaking the metrics gathering to get host metrics (CPU, RAM ...).