From 3bfd194d469c04da7c74a3964bf31ef40605a178 Mon Sep 17 00:00:00 2001 From: Chris Ball Date: Mon, 2 Oct 2023 04:33:16 -0700 Subject: benchmark: notebook wording tweaks --- benchmark/benchmark.ipynb | 87 ++++++++++++++++++++++++++--------------------- 1 file changed, 49 insertions(+), 38 deletions(-) diff --git a/benchmark/benchmark.ipynb b/benchmark/benchmark.ipynb index 7ff90fcd..e4c29f2f 100644 --- a/benchmark/benchmark.ipynb +++ b/benchmark/benchmark.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "code", - "execution_count": 12, + "execution_count": 40, "metadata": {}, "outputs": [], "source": [ @@ -27,7 +27,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 41, "metadata": {}, "outputs": [ { @@ -89,7 +89,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 42, "metadata": {}, "outputs": [ { @@ -371,7 +371,7 @@ "[5 rows x 37 columns]" ] }, - "execution_count": 14, + "execution_count": 42, "metadata": {}, "output_type": "execute_result" } @@ -388,23 +388,14 @@ "source": [ "### Graph prep\n", "\n", - "We're looking for a line graph showing lines for the following cases:\n", - "\n", - "* For each mode:\n", - " * For each target:\n", - " * persistent off, system off\n", - " * persistent on, system off\n", - " * persistent off, system on\n", - " * persistent on, system on\n", - "\n", - "where the x-axis is number of cores, and the y-axis is either afl_execs_per_sec or total_execs_per_sec (I'm not yet sure which is a better metric to use).\n", + "We're looking for a line graph showing lines for each fuzz target, in both singlecore and multicore modes, in each config setting -- where the x-axis is number of cores, and the y-axis is either afl_execs_per_sec or total_execs_per_sec (I'm not yet sure which is a better metric to use).\n", "\n", "But first, a mini test harness by checking that the number of rows matched what we'd intuitively expect:" ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 43, "metadata": {}, "outputs": [], "source": [ @@ -418,7 +409,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 44, "metadata": {}, "outputs": [], "source": [ @@ -470,13 +461,13 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 45, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ - "1234567891011121314151617181920212223242526272829303132333435361x26x51x75x100x125xConfigurationMulticore: Non-persistent mode + kernel configMulticore: Persistent mode/shared memory + kernel configMulticore: Persistent mode/shared memory without kernel configMulticore: afl_execs: Persistent mode/shared memory + kernel configSinglecore: Non-persistent mode + kernel configSinglecore: Persistent mode/shared memory + kernel configFuzzer performanceNumber of parallel fuzzersFuzz target executions per second" + "1234567891011121314151617181920212223242526272829303132333435361x26x51x75x100x125xConfigurationMulticore: Non-persistent mode + kernel configMulticore: Persistent mode/shared memory + kernel configMulticore: Persistent mode/shared memory without kernel configMulticore: afl_execs: Persistent mode/shared memory + kernel configSinglecore: Non-persistent mode + kernel configSinglecore: Persistent mode/shared memory + kernel configFuzzer performanceNumber of parallel fuzzersFuzz target executions per second" ] }, "metadata": {}, @@ -518,7 +509,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 46, "metadata": {}, "outputs": [ { @@ -659,7 +650,7 @@ "5.0 135613.26 " ] }, - "execution_count": 18, + "execution_count": 46, "metadata": {}, "output_type": "execute_result" } @@ -677,7 +668,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 47, "metadata": {}, "outputs": [ { @@ -690,18 +681,25 @@ "#### test-instr vs. test-instr-persist-shmem\n", "\n", "This graph is scaled so that the single-core, non-persistent-mode performance (11038 execs per second) is\n", - "represented as 1.0x. If you build and run a fuzzer without creating a persistent mode harness for it, and without running fuzzers in parallel, this is the performance\n", + "represented as **1.0x**. If you build and run a fuzzer without creating a persistent mode harness for it, and without running fuzzers in parallel, this is the performance\n", "you get on this machine.\n", "\n", "#### Multicore test-instr\n", "\n", - "By running as many parallel fuzzers are there are CPU threads, we can reach 103765 execs per second, which is 9.4x that base speed.\n", + "By running as many parallel fuzzers are there are CPU threads, we can reach 103765 execs per second, which is **9.4x** that base speed.\n", "\n", "#### Persistent mode + shared memory\n", "\n", - "By modifying the harness to use persistent mode as described [here](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#4-persistent-mode),\n", - "we end up with 12.3x base speed. So -- perhaps counter-intuively -- if you have a choice between switching to using multiple cores or rewriting\n", - "the harness to use persistent mode, it is better (at least on this machine) to use persistent mode on a single core, than to use non-persistent mode on every core.\n", + "##### Singlecore\n", + "\n", + "By modifying the harness to use persistent mode with shared memory as described [here](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#4-persistent-mode),\n", + "we end up with **12.3x** base speed. So -- perhaps counter-intuively -- if you have a choice between switching to using multiple cores or rewriting\n", + "the harness to use persistent mode on a single core, it is better (at least on this machine) to use persistent mode on a single core, than to use non-persistent mode on all cores.\n", + "\n", + "##### Multicore\n", + "\n", + "By scaling up that persistent mode with shared memory harness across cores, and with kernel mitigations still turned on (see next section), we get to\n", + "**75.6x** base speed.\n", "\n", "#### Kernel config\n", "\n", @@ -716,6 +714,9 @@ "Or if we want to think in terms of cores, then the average number of execs gained per core in the initial eight is 115588 execs per sec, but the loss due to\n", "mitigations is 368476 execs per sec, which is the averaged performance of 3.2 cores.\n", "\n", + "With kernel mitigations turned off, we reach our highest available total_execs_per_sec speed on this machine, which is **109.0x** higher\n", + "than where we started from.\n", + "\n", "#### afl_execs_per_sec vs. total_execs_per_sec\n", "\n", "* The purple line at the top is measuring `afl_execs_per_sec`. This is afl's own measurement of the speed of each fuzzer process, from the `out/fuzzer/fuzzer_stats` file.\n", @@ -736,7 +737,7 @@ "" ] }, - "execution_count": 19, + "execution_count": 47, "metadata": {}, "output_type": "execute_result" } @@ -765,18 +766,25 @@ "#### test-instr vs. test-instr-persist-shmem\n", "\n", "This graph is scaled so that the single-core, non-persistent-mode performance ({int(singlecore_base_execs)} execs per second) is\n", - "represented as 1.0x. If you build and run a fuzzer without creating a persistent mode harness for it, and without running fuzzers in parallel, this is the performance\n", + "represented as **1.0x**. If you build and run a fuzzer without creating a persistent mode harness for it, and without running fuzzers in parallel, this is the performance\n", "you get on this machine.\n", "\n", "#### Multicore test-instr\n", "\n", - "By running as many parallel fuzzers are there are CPU threads, we can reach {int(multicore_base_max_execs)} execs per second, which is {factor_for_execs(multicore_base_max_execs)}x that base speed.\n", + "By running as many parallel fuzzers are there are CPU threads, we can reach {int(multicore_base_max_execs)} execs per second, which is **{factor_for_execs(multicore_base_max_execs)}x** that base speed.\n", "\n", "#### Persistent mode + shared memory\n", "\n", - "By modifying the harness to use persistent mode as described [here](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#4-persistent-mode),\n", - "we end up with {factor_for_execs(singlecore_persist_execs)}x base speed. So -- perhaps counter-intuively -- if you have a choice between switching to using multiple cores or rewriting\n", - "the harness to use persistent mode, it is better (at least on this machine) to use persistent mode on a single core, than to use non-persistent mode on every core.\n", + "##### Singlecore\n", + "\n", + "By modifying the harness to use persistent mode with shared memory as described [here](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#4-persistent-mode),\n", + "we end up with **{factor_for_execs(singlecore_persist_execs)}x** base speed. So -- perhaps counter-intuively -- if you have a choice between switching to using multiple cores or rewriting\n", + "the harness to use persistent mode on a single core, it is better (at least on this machine) to use persistent mode on a single core, than to use non-persistent mode on all cores.\n", + "\n", + "##### Multicore\n", + "\n", + "By scaling up that persistent mode with shared memory harness across cores, and with kernel mitigations still turned on (see next section), we get to\n", + "**{factor_for_execs(multicore_max_execs_mitigations_on)}x** base speed.\n", "\n", "#### Kernel config\n", "\n", @@ -791,6 +799,9 @@ "Or if we want to think in terms of cores, then the average number of execs gained per core in the initial eight is {int(multicore_avg_gain_per_core)} execs per sec, but the loss due to\n", "mitigations is {mitigations_off_increase} execs per sec, which is the averaged performance of {round(mitigations_off_increase / multicore_avg_gain_per_core, 1)} cores.\n", "\n", + "With kernel mitigations turned off, we reach our highest available total_execs_per_sec speed on this machine, which is **{factor_for_execs(multicore_max_execs_mitigations_off)}x** higher\n", + "than where we started from.\n", + "\n", "#### afl_execs_per_sec vs. total_execs_per_sec\n", "\n", "* The purple line at the top is measuring `afl_execs_per_sec`. This is afl's own measurement of the speed of each fuzzer process, from the `out/fuzzer/fuzzer_stats` file.\n", @@ -820,7 +831,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 48, "metadata": {}, "outputs": [ { @@ -955,7 +966,7 @@ "149 6.15 " ] }, - "execution_count": 20, + "execution_count": 48, "metadata": {}, "output_type": "execute_result" } @@ -967,7 +978,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 49, "metadata": {}, "outputs": [ { @@ -1029,7 +1040,7 @@ "153 True Multicore: afl_execs: Persistent mode/shared m... " ] }, - "execution_count": 21, + "execution_count": 49, "metadata": {}, "output_type": "execute_result" } @@ -1041,13 +1052,13 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 50, "metadata": {}, "outputs": [ { "data": { "image/svg+xml": [ - "510152025303540455055606570758085909510010511011512012513013514014515015516016517017518018519019520010x36x62x89x115x141xConfigurationMulticore: Persistent mode/shared memory + kernel configMulticore: afl_execs: Persistent mode/shared memory + kernel configFuzzer performanceNumber of parallel fuzzersFuzz target executions per second" + "510152025303540455055606570758085909510010511011512012513013514014515015516016517017518018519019520010x36x62x89x115x141xConfigurationMulticore: Persistent mode/shared memory + kernel configMulticore: afl_execs: Persistent mode/shared memory + kernel configFuzzer performanceNumber of parallel fuzzersFuzz target executions per second" ] }, "metadata": {}, -- cgit 1.4.1