From b9db6b1254c9bf3a47c171bb96468628e9bd00f2 Mon Sep 17 00:00:00 2001 From: Chris Ball Date: Mon, 2 Oct 2023 03:23:09 -0700 Subject: benchmark: add a README, lower default runs from 5 to 3 --- benchmark/README.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 benchmark/README.md (limited to 'benchmark/README.md') diff --git a/benchmark/README.md b/benchmark/README.md new file mode 100644 index 00000000..66f7f59e --- /dev/null +++ b/benchmark/README.md @@ -0,0 +1,46 @@ +# American Fuzzy Lop plus plus (AFL++) + +## benchmarking + +This directory contains benchmarking tools that allow you to compare one machine +with another in terms of raw ability to execute a fuzzing target repeatedly. + +To achieve this, we use a sample program ("test-instr.c") where each path is +equally likely, supply it a single seed, and tell AFL to exit after one run of +deterministic mutations against that seed. + +Usage: + +``` +cd aflplusplus/benchmark +python3 benchmark.py + [*] Using 16 fuzzers for multicore fuzzing (use --fuzzers to override) + [*] Ready, starting benchmark... + [*] Compiling the test-instr-persist-shmem fuzzing harness for the benchmark to use. + [*] multicore test-instr-persist-shmem run 1 of 3, execs/s: 846065.81 + [*] multicore test-instr-persist-shmem run 2 of 3, execs/s: 849694.03 + [*] multicore test-instr-persist-shmem run 3 of 3, execs/s: 850757.52 + [*] Average AFL execs/sec for this test across all runs was: 848839.12 + [*] Average total execs/sec for this test across all runs was: 833138.28 + [*] Results have been written to benchmark-results.jsonl +``` + +By default, the script will use a number of parallel fuzzers equal to your +available CPUs/threads (change with `--fuzzers`), and will perform each test +three times and average the result (change with `--runs`). + +The script will use multicore fuzzing instead of singlecore by default (change +with `--mode singlecore`) and use a persistent-mode shared memory harness for +optimal speed (change with `--target test-instr`). + +Each run writes results to [benchmark-results.jsonl](benchmark-results.jsonl) +in [JSON Lines](https://jsonlines.org/) format, ready to be pulled in to other +tools such as [jq -cs](https://jqlang.github.io/jq/) or +[pandas](https://pandas.pydata.org/) for analysis. + +## Data analysis + +There is sample data in [benchmark-results.jsonl](benchmark-results.jsonl), and +a Jupyter notebook for exploring the results and suggesting their meaning at +[benchmark.ipynb](benchmark.ipynb). + -- cgit 1.4.1 From f2cbcdf3ff7349ab505e1fcebc3242c9252f2176 Mon Sep 17 00:00:00 2001 From: Chris Ball Date: Sun, 19 Nov 2023 15:10:23 -0800 Subject: benchmark: update README --- benchmark/README.md | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) (limited to 'benchmark/README.md') diff --git a/benchmark/README.md b/benchmark/README.md index 66f7f59e..e37abad2 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -9,20 +9,22 @@ To achieve this, we use a sample program ("test-instr.c") where each path is equally likely, supply it a single seed, and tell AFL to exit after one run of deterministic mutations against that seed. -Usage: +Usage example: ``` cd aflplusplus/benchmark python3 benchmark.py - [*] Using 16 fuzzers for multicore fuzzing (use --fuzzers to override) [*] Ready, starting benchmark... [*] Compiling the test-instr-persist-shmem fuzzing harness for the benchmark to use. - [*] multicore test-instr-persist-shmem run 1 of 3, execs/s: 846065.81 - [*] multicore test-instr-persist-shmem run 2 of 3, execs/s: 849694.03 - [*] multicore test-instr-persist-shmem run 3 of 3, execs/s: 850757.52 - [*] Average AFL execs/sec for this test across all runs was: 848839.12 - [*] Average total execs/sec for this test across all runs was: 833138.28 - [*] Results have been written to benchmark-results.jsonl + [*] singlecore test-instr-persist-shmem run 1 of 2, execs/s: 124883.62 + [*] singlecore test-instr-persist-shmem run 2 of 2, execs/s: 126704.93 + [*] Average execs/sec for this test across all runs was: 125794.28 + [*] Using 16 fuzzers for multicore fuzzing (use --fuzzers to override). + [*] multicore test-instr-persist-shmem run 1 of 2, execs/s: 1179822.66 + [*] multicore test-instr-persist-shmem run 2 of 2, execs/s: 1175584.09 + [*] Average execs/sec for this test across all runs was: 1177703.38 + [*] Results have been written to the benchmark-results.jsonl file. + [*] Results have been written to the COMPARISON file. ``` By default, the script will use a number of parallel fuzzers equal to your @@ -33,6 +35,9 @@ The script will use multicore fuzzing instead of singlecore by default (change with `--mode singlecore`) and use a persistent-mode shared memory harness for optimal speed (change with `--target test-instr`). +Feel free to submit the resulting line for your CPU added to the COMPARISON +file back to aflplusplus in a pull request. + Each run writes results to [benchmark-results.jsonl](benchmark-results.jsonl) in [JSON Lines](https://jsonlines.org/) format, ready to be pulled in to other tools such as [jq -cs](https://jqlang.github.io/jq/) or -- cgit 1.4.1