diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/Changelog.md | 19 | ||||
| -rw-r--r-- | docs/FAQ.md | 95 | ||||
| -rw-r--r-- | docs/binaryonly_fuzzing.md | 12 | ||||
| -rw-r--r-- | docs/custom_mutators.md | 7 | ||||
| -rw-r--r-- | docs/env_variables.md | 29 | ||||
| -rw-r--r-- | docs/parallel_fuzzing.md | 113 |
6 files changed, 154 insertions, 121 deletions
diff --git a/docs/Changelog.md b/docs/Changelog.md index cb6e14b8..d1ee9656 100644 --- a/docs/Changelog.md +++ b/docs/Changelog.md @@ -10,17 +10,24 @@ sending a mail to <afl-users+subscribe@googlegroups.com>. ### Version ++2.67d (develop) + - added the GSoC excellent afl++ grammar mutator by Shengtuo to our + custom_mutators/ (see custom_mutators/README.md) - or get it here: + https://github.com/AFLplusplus/Grammar-Mutator + - a few QOL changes for Apple and its outdated gmake - afl-fuzz: - Fix for auto dictionary entries found during fuzzing to not throw out a -x dictionary - added total execs done to plot file - - AFL_MAX_DET_EXTRAS env variable added to control the amount of deterministic - dict entries without recompiling. - - AFL_FORKSRV_INIT_TMOUT env variable added to control the time to wait for - the forkserver to come up without the need to increase the overall timeout. + - AFL_MAX_DET_EXTRAS env variable added to control the amount of + deterministic dict entries without recompiling. + - AFL_FORKSRV_INIT_TMOUT env variable added to control the time to wait + for the forkserver to come up without the need to increase the overall + timeout. + - bugfix for cmplog that results in a heap overflow based on target data + (thanks to the magma team for reporting!) - custom mutators: - - added afl_custom_fuzz_count/fuzz_count function to allow specifying the - number of fuzz attempts for custom_fuzz + - added afl_custom_fuzz_count/fuzz_count function to allow specifying + the number of fuzz attempts for custom_fuzz - llvm_mode: - Ported SanCov to LTO, and made it the default for LTO. better instrumentation locations diff --git a/docs/FAQ.md b/docs/FAQ.md index 93a87a72..df5cc79c 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -4,11 +4,11 @@ * [What is the difference between afl and afl++?](#what-is-the-difference-between-afl-and-afl) * [How to improve the fuzzing speed?](#how-to-improve-the-fuzzing-speed) - * [How do I fuzz a network service?](#how-to-fuzz-a-network-service) - * [How do I fuzz a GUI program?](#how-to-fuzz-a-gui-program) + * [How do I fuzz a network service?](#how-do-i-fuzz-a-network-service) + * [How do I fuzz a GUI program?](#how-do-i-fuzz-a-gui-program) * [What is an edge?](#what-is-an-edge) * [Why is my stability below 100%?](#why-is-my-stability-below-100) - * [How can I improve the stability value](#how-can-i-improve-the-stability-value) + * [How can I improve the stability value?](#how-can-i-improve-the-stability-value) If you find an interesting or important question missing, submit it via [https://github.com/AFLplusplus/AFLplusplus/issues](https://github.com/AFLplusplus/AFLplusplus/issues) @@ -18,51 +18,54 @@ If you find an interesting or important question missing, submit it via American Fuzzy Lop (AFL) was developed by Michał "lcamtuf" Zalewski starting in 2013/2014, and when he left Google end of 2017 he stopped developing it. -At the end of 2019 the Google fuzzing team took over maintance of AFL, however -it is only accepting PR from the community and is not developing enhancements +At the end of 2019 the Google fuzzing team took over maintenance of AFL, however +it is only accepting PRs from the community and is not developing enhancements anymore. -In the second quarter of 2019, 1 1/2 years after no further development of -AFL had happened and it became clear there would be none coming, afl++ -was born, where initially first community patches were collected and applied -for bugs and enhancements. Then from various AFL spin-offs - mostly academic +In the second quarter of 2019, 1 1/2 year later when no further development of +AFL had happened and it became clear there would none be coming, afl++ +was born, where initially community patches were collected and applied +for bug fixes and enhancements. Then from various AFL spin-offs - mostly academic research - features were integrated. This already resulted in a much advanced AFL. Until the end of 2019 the afl++ team had grown to four active developers which -then implemented their own research and feature, making it now by far the most +then implemented their own research and features, making it now by far the most flexible and feature rich guided fuzzer available as open source. And in independent fuzzing benchmarks it is one of the best fuzzers available, e.g. [Fuzzbench Report](https://www.fuzzbench.com/reports/2020-08-03/index.html) -## How to improve the fuzzing speed +## How to improve the fuzzing speed? - 1. use [llvm_mode](docs/llvm_mode/README.md): afl-clang-lto (llvm >= 11) or afl-clang-fast (llvm >= 9 recommended) + 1. Use [llvm_mode](docs/llvm_mode/README.md): afl-clang-lto (llvm >= 11) or afl-clang-fast (llvm >= 9 recommended) 2. Use [persistent mode](llvm_mode/README.persistent_mode.md) (x2-x20 speed increase) 3. Use the [afl++ snapshot module](https://github.com/AFLplusplus/AFL-Snapshot-LKM) (x2 speed increase) - 4. If you do not use shmem persistent mode, use `AFL_TMPDIR` to point the input file on a tempfs location, see [docs/env_variables.md](docs/env_variables.md) - 5. Improve kernel performance: modify `/etc/default/grub`, set `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then `update-grub` and `reboot` (warning: makes the system more insecure) + 4. If you do not use shmem persistent mode, use `AFL_TMPDIR` to put the input file directory on a tempfs location, see [docs/env_variables.md](docs/env_variables.md) + 5. Improve Linux kernel performance: modify `/etc/default/grub`, set `GRUB_CMDLINE_LINUX_DEFAULT="ibpb=off ibrs=off kpti=off l1tf=off mds=off mitigations=off no_stf_barrier noibpb noibrs nopcid nopti nospec_store_bypass_disable nospectre_v1 nospectre_v2 pcid=off pti=off spec_store_bypass_disable=off spectre_v2=off stf_barrier=off"`; then `update-grub` and `reboot` (warning: makes the system more insecure) 6. Running on an `ext2` filesystem with `noatime` mount option will be a bit faster than on any other journaling filesystem 7. Use your cores! [README.md:3.b) Using multiple cores/threads](../README.md#b-using-multiple-coresthreads) ## How do I fuzz a network service? -The short answer is - you cannot, at least "out of the box". +The short answer is - you cannot, at least not "out of the box". -Using network has a slow-down of x10-20 on the fuzzing speed, does not scale, -and finally usually it is more than one initial data packet but a back-and-forth -which is totally unsupported by most coverage aware fuzzers. +Using a network channel is inadequate for several reasons: +- it has a slow-down of x10-20 on the fuzzing speed +- it does not scale to multiple connections, +- instead of one initial data packet often a back-and-forth +interplay of packets is needed for stateful protocols +(which is totally unsupported by most coverage aware fuzzers). The established method to fuzz network services is to modify the source code to read from a file or stdin (fd 0) (or even faster via shared memory, combine this with persistent mode [llvm_mode/README.persistent_mode.md](llvm_mode/README.persistent_mode.md) and you have a performance gain of x10 instead of a performance loss of over -x10 - that is a x100 difference! +x10 - that is a x100 difference!). If modifying the source is not an option (e.g. because you only have a binary and perform binary fuzzing) you can also use a shared library with AFL_PRELOAD -to emulate the network. This is also much faster than network would be. -See [examples/socket_fuzzing/](../examples/socket_fuzzing/) +to emulate the network. This is also much faster than the real network would be. +See [examples/socket_fuzzing/](../examples/socket_fuzzing/). There is an outdated afl++ branch that implements networking if you are desperate though: [https://github.com/AFLplusplus/AFLplusplus/tree/networking](https://github.com/AFLplusplus/AFLplusplus/tree/networking) - @@ -73,7 +76,7 @@ which allows you to define network state with different type of data packets. If the GUI program can read the fuzz data from a file (via the command line, a fixed location or via an environment variable) without needing any user -interaction then then yes. +interaction then it would be suitable for fuzzing. Otherwise it is not possible without modifying the source code - which is a very good idea anyway as the GUI functionality is a huge CPU/time overhead @@ -82,13 +85,13 @@ for the fuzzing. So create a new `main()` that just reads the test case and calls the functionality for processing the input that the GUI program is using. -## What is an "edge" +## What is an "edge"? A program contains `functions`, `functions` contain the compiled machine code. The compiled machine code in a `function` can be in a single or many `basic blocks`. A `basic block` is the largest possible number of subsequent machine code -instructions that runs independent, meaning it does not split up to different -locations nor is it jumped into it from a different location: +instructions that has exactly one entry (at the beginning) and runs linearly without +branching or jumping to other addresses (except at the end). ``` function() { A: @@ -98,7 +101,7 @@ function() { if (x) goto C; else goto D; C: some code - goto D + goto E D: some code goto B @@ -108,7 +111,7 @@ function() { ``` Every code block between two jump locations is a `basic block`. -An `edge` is then the unique relationship between two `basic blocks` (from the +An `edge` is then the unique relationship between two directly connected `basic blocks` (from the code example above): ``` Block A @@ -124,7 +127,7 @@ code example above): ``` Every line between two blocks is an `edge`. -## Why is my stability below 100% +## Why is my stability below 100%? Stability is measured by how many percent of the edges in the target are "stable". Sending the same input again and again should take the exact same @@ -132,37 +135,37 @@ path through the target every time. If that is the case, the stability is 100%. If however randomness happens, e.g. a thread reading other external data, reaction to timing, etc. then in some of the re-executions with the same data -the result in the edge information will be different accross runs. +the edge coverage result will be different accross runs. Those edges that change are then flagged "unstable". The more "unstable" edges, the more difficult for afl++ to identify valid new paths. A value above 90% is usually fine and a value above 80% is also still ok, and -even above 20% can still result in successful finds of bugs. -However, it is recommended that below 90% or 80% you should take measures to -improve the stability. +even a value above 20% can still result in successful finds of bugs. +However, it is recommended that for values below 90% or 80% you should take +countermeasures to improve stability. -## How can I improve the stability value +## How can I improve the stability value? -For fuzzing a 100% stable target that covers all edges is the best. +For fuzzing a 100% stable target that covers all edges is the best case. A 90% stable target that covers all edges is however better than a 100% stable target that ignores 10% of the edges. With instability you basically have a partial coverage loss on an edge, with -ignore you have a full loss on that edge. +ignored functions you have a full loss on that edges. There are functions that are unstable, but also provide value to coverage, eg init functions that use fuzz data as input for example. -If however it is a function that has nothing to do with the input data is the -source, e.g. checking jitter, or is a hash map function etc. then it should -not be instrumented. +If however a function that has nothing to do with the input data is the +source of instability, e.g. checking jitter, or is a hash map function etc. +then it should not be instrumented. -To be able to make this decision the following process will allow you to -identify the functions with variable edges so you can make this decision. +To be able to exclude these functions (based on AFL++'s measured stability) +the following process will allow to identify functions with variable edges. -Four steps are required to do this and requires quite some knowledge of -coding and/or disassembly and it is only effectively possible with +Four steps are required to do this and it also requires quite some knowledge +of coding and/or disassembly and is effectively possible only with afl-clang-fast PCGUARD and afl-clang-lto LTO instrumentation. 1. First step: Identify which edge ID numbers are unstable @@ -171,7 +174,7 @@ afl-clang-fast PCGUARD and afl-clang-lto LTO instrumentation. The out/fuzzer_stats file will then show the edge IDs that were identified as unstable. - 2. Second step: Find the responsible function. + 2. Second step: Find the responsible function(s). a) For LTO instrumented binaries this can be documented during compile time, just set `export AFL_LLVM_DOCUMENT_IDS=/path/to/a/file`. @@ -191,20 +194,20 @@ afl-clang-fast PCGUARD and afl-clang-lto LTO instrumentation. and set a write breakpoint to that address (`watch 0x.....`). c) in all other instrumentation types this is not possible. So just - recompile with the the two mentioned above. This is just for + recompile with the two mentioned above. This is just for identifying the functions that have unstable edges. 3. Third step: create a text file with the filenames/functions Identify which source code files contain the functions that you need to remove from instrumentation, or just specify the functions you want to - skip instrumenting. Note that optimization might inline functions! + skip for instrumentation. Note that optimization might inline functions! Simply follow this document on how to do this: [llvm_mode/README.instrument_list.md](llvm_mode/README.instrument_list.md) If PCGUARD is used, then you need to follow this guide (needs llvm 12+!): [http://clang.llvm.org/docs/SanitizerCoverage.html#partially-disabling-instrumentation](http://clang.llvm.org/docs/SanitizerCoverage.html#partially-disabling-instrumentation) - Only deny those functions from instrumentation that provide no value + Only exclude those functions from instrumentation that provide no value for coverage - that is if it does not process any fuzz data directly or indirectly (e.g. hash maps, thread management etc.). If however a function directly or indirectly handles fuzz data then you diff --git a/docs/binaryonly_fuzzing.md b/docs/binaryonly_fuzzing.md index a3d3330f..cb1288ef 100644 --- a/docs/binaryonly_fuzzing.md +++ b/docs/binaryonly_fuzzing.md @@ -6,14 +6,14 @@ However, if there is only the binary program and no source code available, then standard `afl-fuzz -n` (non-instrumented mode) is not effective. - The following is a description of how these binaries can be fuzzed with afl++ + The following is a description of how these binaries can be fuzzed with afl++. ## TL;DR: qemu_mode in persistent mode is the fastest - if the stability is high enough. Otherwise try retrowrite, afl-dyninst and if these - fail too then standard qemu_mode with AFL_ENTRYPOINT to where you need it. + fail too then try standard qemu_mode with AFL_ENTRYPOINT to where you need it. If your target is a library use examples/afl_frida/. @@ -29,10 +29,10 @@ The speed decrease is at about 50%. However various options exist to increase the speed: - - using AFL_ENTRYPOINT to move the forkserver to a later basic block in + - using AFL_ENTRYPOINT to move the forkserver entry to a later basic block in the binary (+5-10% speed) - using persistent mode [qemu_mode/README.persistent.md](../qemu_mode/README.persistent.md) - this will result in 150-300% overall speed - so 3-8x the original + this will result in 150-300% overall speed increase - so 3-8x the original qemu_mode speed! - using AFL_CODE_START/AFL_CODE_END to only instrument specific parts @@ -104,7 +104,7 @@ ## RETROWRITE - If you have an x86/x86_64 binary that still has it's symbols, is compiled + If you have an x86/x86_64 binary that still has its symbols, is compiled with position independant code (PIC/PIE) and does not use most of the C++ features then the retrowrite solution might be for you. It decompiles to ASM files which can then be instrumented with afl-gcc. @@ -148,7 +148,7 @@ ## CORESIGHT Coresight is ARM's answer to Intel's PT. - There is no implementation so far which handle coresight and getting + There is no implementation so far which handles coresight and getting it working on an ARM Linux is very difficult due to custom kernel building on embedded systems is difficult. And finding one that has coresight in the ARM chip is difficult too. diff --git a/docs/custom_mutators.md b/docs/custom_mutators.md index 75dbea21..a128f587 100644 --- a/docs/custom_mutators.md +++ b/docs/custom_mutators.md @@ -94,8 +94,11 @@ def queue_new_entry(filename_new_queue, filename_orig_queue): - `fuzz_count` (optional): - This method can be used to instruct afl-fuzz how often to perform a fuzz - attempt on this input data. + When a queue entry is selected to be fuzzed, afl-fuzz selects the number + of fuzzing attempts with this input based on a few factors. + If however the custom mutator wants to set this number instead on how often + it is called for a specific queue entry, use this function. + This function in mostly useful if **not** `AFL_CUSTOM_MUTATOR_ONLY` is used. - `fuzz` (optional): diff --git a/docs/env_variables.md b/docs/env_variables.md index e8129a3f..c47d10e8 100644 --- a/docs/env_variables.md +++ b/docs/env_variables.md @@ -10,8 +10,8 @@ Because they can't directly accept command-line options, the compile-time tools make fairly broad use of environmental variables: - - Most afl tools do not print any ouput if stout/stderr are redirected. - If you want to have the output into a file then set the AFL_DEBUG + - Most afl tools do not print any output if stdout/stderr are redirected. + If you want to save the output in a file then set the AFL_DEBUG environment variable. This is sadly necessary for various build processes which fail otherwise. @@ -44,7 +44,7 @@ tools make fairly broad use of environmental variables: you instrument hand-written assembly when compiling clang code by plugging a normalizer into the chain. (There is no equivalent feature for GCC.) - - Setting AFL_INST_RATIO to a percentage between 0 and 100% controls the + - Setting AFL_INST_RATIO to a percentage between 0% and 100% controls the probability of instrumenting every branch. This is (very rarely) useful when dealing with exceptionally complex programs that saturate the output bitmap. Examples include v8, ffmpeg, and perl. @@ -88,7 +88,7 @@ of the settings discussed in section #1, with the exception of: - TMPDIR and AFL_KEEP_ASSEMBLY, since no temporary assembly files are created. - - AFL_INST_RATIO, as we by default collision free instrumentation is used. + - AFL_INST_RATIO, as we by default use collision free instrumentation. Then there are a few specific features that are only available in llvm_mode: @@ -121,7 +121,7 @@ Then there are a few specific features that are only available in llvm_mode: None of the following options are necessary to be used and are rather for manual use (which only ever the author of this LTO implementation will use). - These are used if several seperated instrumentation are performed which + These are used if several seperated instrumentations are performed which are then later combined. - AFL_LLVM_DOCUMENT_IDS=file will document to a file which edge ID was given @@ -200,7 +200,7 @@ Then there are a few specific features that are only available in llvm_mode: ### INSTRUMENT LIST (selectively instrument files and functions) - This feature allows selectively instrumentation of the source + This feature allows selective instrumentation of the source - Setting AFL_LLVM_ALLOWLIST or AFL_LLVM_DENYLIST with a filenames and/or function will only instrument (or skip) those files that match the names @@ -369,13 +369,6 @@ checks or alter some of the more exotic semantics of the tool: for an existing out folder, even if a different `-i` was provided. Without this setting, afl-fuzz will refuse execution for a long-fuzzed out dir. - - Setting AFL_MAX_DET_EXRAS will change the threshold at what number of elements - in the `-x` dictionary and LTO autodict (combined) the probabilistic mode will - kick off. In probabilistic mode, not all dictionary entires will be used all - of the times for fuzzing mutations to not make fuzzing slower by it. - The default count is `200` element. So for the 200 + 1st element, there is a - 1 in 201 chance, that one of the dictionary entry will not be used directly. - - Setting AFL_NO_FORKSRV disables the forkserver optimization, reverting to fork + execve() call for every tested input. This is useful mostly when working with unruly libraries that create threads or do other crazy @@ -384,6 +377,16 @@ checks or alter some of the more exotic semantics of the tool: Note that this setting inhibits some of the user-friendly diagnostics normally done when starting up the forkserver and causes a pretty significant performance drop. + + - Setting AFL_MAX_DET_EXTRAS changes the count of dictionary entries/extras + (default 200), after which the entries will be used probabilistically. + So, if the dict/extras file (`-x`) contains more tokens than this threshold, + not all of the tokens will be used in each fuzzing step, every time. + Instead, there is a chance that the entry will be skipped during fuzzing. + This makes sure that the fuzzer doesn't spend all its time only inserting + the extras, but will still do other mutations. However, it decreases the + likelihood for each token to be inserted, before the next queue entry is fuzzed. + Either way, all tokens will be used eventually, in a longer fuzzing campaign. - Outdated environment variables that are that not supported anymore: AFL_DEFER_FORKSRV diff --git a/docs/parallel_fuzzing.md b/docs/parallel_fuzzing.md index 2ab1466c..bf57ace8 100644 --- a/docs/parallel_fuzzing.md +++ b/docs/parallel_fuzzing.md @@ -10,8 +10,8 @@ n-core system, you can almost always run around n concurrent fuzzing jobs with virtually no performance hit (you can use the afl-gotcpu tool to make sure). In fact, if you rely on just a single job on a multi-core system, you will -be underutilizing the hardware. So, parallelization is usually the right -way to go. +be underutilizing the hardware. So, parallelization is always the right way to +go. When targeting multiple unrelated binaries or using the tool in "non-instrumented" (-n) mode, it is perfectly fine to just start up several @@ -65,22 +65,7 @@ still perform deterministic checks; while the secondary instances will proceed straight to random tweaks. Note that you must always have one -M main instance! - -Note that running multiple -M instances is wasteful, although there is an -experimental support for parallelizing the deterministic checks. To leverage -that, you need to create -M instances like so: - -``` -./afl-fuzz -i testcase_dir -o sync_dir -M mainA:1/3 [...] -./afl-fuzz -i testcase_dir -o sync_dir -M mainB:2/3 [...] -./afl-fuzz -i testcase_dir -o sync_dir -M mainC:3/3 [...] -``` - -...where the first value after ':' is the sequential ID of a particular main -instance (starting at 1), and the second value is the total number of fuzzers to -distribute the deterministic fuzzing across. Note that if you boot up fewer -fuzzers than indicated by the second number passed to -M, you may end up with -poor coverage. +Running multiple -M instances is wasteful! You can also monitor the progress of your jobs from the command line with the provided afl-whatsup tool. When the instances are no longer finding new paths, @@ -99,61 +84,88 @@ example may be: This is not a concern if you use @@ without -f and let afl-fuzz come up with the file name. -## 3) Syncing with non-afl fuzzers or independant instances +## 3) Multiple -M mains + + +There is support for parallelizing the deterministic checks. +This is only needed where + + 1. many new paths are found fast over a long time and it looks unlikely that + main node will ever catch up, and + 2. deterministic fuzzing is actively helping path discovery (you can see this + in the main node for the first for lines in the "fuzzing strategy yields" + section. If the ration `found/attemps` is high, then it is effective. It + most commonly isn't.) + +Only if both are true it is beneficial to have more than one main. +You can leverage this by creating -M instances like so: + +``` +./afl-fuzz -i testcase_dir -o sync_dir -M mainA:1/3 [...] +./afl-fuzz -i testcase_dir -o sync_dir -M mainB:2/3 [...] +./afl-fuzz -i testcase_dir -o sync_dir -M mainC:3/3 [...] +``` + +... where the first value after ':' is the sequential ID of a particular main +instance (starting at 1), and the second value is the total number of fuzzers to +distribute the deterministic fuzzing across. Note that if you boot up fewer +fuzzers than indicated by the second number passed to -M, you may end up with +poor coverage. + +## 4) Syncing with non-afl fuzzers or independant instances A -M main node can be told with the `-F other_fuzzer_queue_directory` option to sync results from other fuzzers, e.g. libfuzzer or honggfuzz. Only the specified directory will by synced into afl, not subdirectories. -The specified directories do not need to exist yet at the start of afl. +The specified directory does not need to exist yet at the start of afl. -## 4) Multi-system parallelization +The `-F` option can be passed to the main node several times. + +## 5) Multi-system parallelization The basic operating principle for multi-system parallelization is similar to the mechanism explained in section 2. The key difference is that you need to write a simple script that performs two actions: - Uses SSH with authorized_keys to connect to every machine and retrieve - a tar archive of the /path/to/sync_dir/<fuzzer_id>/queue/ directories for - every <fuzzer_id> local to the machine. It's best to use a naming scheme - that includes host name in the fuzzer ID, so that you can do something - like: + a tar archive of the /path/to/sync_dir/<main_node(s)> directory local to + the machine. + It is best to use a naming scheme that includes host name and it's being + a main node (e.g. main1, main2) in the fuzzer ID, so that you can do + something like: ```sh - for s in {1..10}; do - ssh user@host${s} "tar -czf - sync/host${s}_fuzzid*/[qf]*" >host${s}.tgz + for host in `cat HOSTLIST`; do + ssh user@$host "tar -czf - sync/$host_main*/" > $host.tgz done ``` - Distributes and unpacks these files on all the remaining machines, e.g.: ```sh - for s in {1..10}; do - for d in {1..10}; do - test "$s" = "$d" && continue - ssh user@host${d} 'tar -kxzf -' <host${s}.tgz + for srchost in `cat HOSTLIST`; do + for dsthost in `cat HOSTLIST`; do + test "$srchost" = "$dsthost" && continue + ssh user@$srchost 'tar -kxzf -' < $dsthost.tgz done done ``` -There is an example of such a script in examples/distributed_fuzzing/; -you can also find a more featured, experimental tool developed by -Martijn Bogaard at: - - https://github.com/MartijnB/disfuzz-afl - -Another client-server implementation from Richo Healey is: +There is an example of such a script in examples/distributed_fuzzing/. - https://github.com/richo/roving +There are other (older) more featured, experimental tools: + * https://github.com/richo/roving + * https://github.com/MartijnB/disfuzz-afl -Note that these third-party tools are unsafe to run on systems exposed to the -Internet or to untrusted users. +However these do not support syncing just main nodes (yet). When developing custom test case sync code, there are several optimizations to keep in mind: - The synchronization does not have to happen very often; running the - task every 30 minutes or so may be perfectly fine. + task every 60 minutes or even less often at later fuzzing stages is + fine - There is no need to synchronize crashes/ or hangs/; you only need to copy over queue/* (and ideally, also fuzzer_stats). @@ -179,19 +191,24 @@ to keep in mind: - You do not want a "main" instance of afl-fuzz on every system; you should run them all with -S, and just designate a single process somewhere within the fleet to run with -M. + + - Syncing is only necessary for the main nodes on a system. It is possible + to run main-less with only secondaries. However then you need to find out + which secondary took over the temporary role to be the main node. Look for + the `is_main_node` file in the fuzzer directories, eg. `sync-dir/hostname-*/is_main_node` It is *not* advisable to skip the synchronization script and run the fuzzers directly on a network filesystem; unexpected latency and unkillable processes in I/O wait state can mess things up. -## 5) Remote monitoring and data collection +## 6) Remote monitoring and data collection You can use screen, nohup, tmux, or something equivalent to run remote instances of afl-fuzz. If you redirect the program's output to a file, it will automatically switch from a fancy UI to more limited status reports. There is -also basic machine-readable information always written to the fuzzer_stats file -in the output directory. Locally, that information can be interpreted with -afl-whatsup. +also basic machine-readable information which is always written to the +fuzzer_stats file in the output directory. Locally, that information can be +interpreted with afl-whatsup. In principle, you can use the status screen of the main (-M) instance to monitor the overall fuzzing progress and decide when to stop. In this @@ -208,7 +225,7 @@ Keep in mind that crashing inputs are *not* automatically propagated to the main instance, so you may still want to monitor for crashes fleet-wide from within your synchronization or health checking scripts (see afl-whatsup). -## 6) Asymmetric setups +## 7) Asymmetric setups It is perhaps worth noting that all of the following is permitted: @@ -224,7 +241,7 @@ It is perhaps worth noting that all of the following is permitted: the discovered test cases can have synergistic effects and improve the overall coverage. - (In this case, running one -M instance per each binary is a good plan.) + (In this case, running one -M instance per target is necessary.) - Having some of the fuzzers invoke the binary in different ways. For example, 'djpeg' supports several DCT modes, configurable with |