push to stable (#1967)

* Output afl-clang-fast stuffs only if necessary (#1912)

* afl-cc header

* afl-cc common declarations

 - Add afl-cc-state.c
 - Strip includes, find_object, debug/be_quiet/have_*/callname setting from afl-cc.c
 - Use debugf_args in main
 - Modify execvp stuffs to fit new aflcc struct

* afl-cc show usage

* afl-cc mode selecting

1. compiler_mode by callname in argv[0]
2. compiler_mode by env "AFL_CC_COMPILER"
3. compiler_mode/instrument_mode by command line options "--afl-..."
4. instrument_mode/compiler_mode by various env vars including "AFL_LLVM_INSTRUMENT"
5. final checking steps
6. print "... - mode: %s-%s\n"
7. determine real argv[0] according to compiler_mode

* afl-cc macro defs

* afl-cc linking behaviors

* afl-cc fsanitize behaviors

* afl-cc misc

* afl-cc body update

* afl-cc all-in-one

formated with custom-format.py

* nits

---------

Co-authored-by: vanhauser-thc <vh@thc.org>

* changelog

* update grammar mutator

* lto llvm 12+

* docs(custom_mutators): fix missing ':' (#1953)

* Fix broken LTO mode and response file support (#1948)

* Strip `-Wl,-no-undefined` during compilation (#1952)

Make the compiler wrapper stripping `-Wl,-no-undefined` in addition to `-Wl,--no-undefined`.
Both versions of the flag are accepted by clang and, therefore, used by building systems in the wild (e.g., samba will not build without this fix).

* Remove dead code in write_to_testcase (#1955)

The custom_mutators_count check in if case is duplicate with if condition.
The else case is custom_mutators_count == 0, neither custom_mutator_list iteration nor sent check needed.

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>

* update qemuafl

* WIP: Add ability to generate drcov trace using QEMU backend (#1956)

* Document new drcov QEMU plugin

* Add link to lightkeeper for QEMU drcov file loading

---------

Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>

* code format

* changelog

* sleep on uid != 0 afl-system-config

* fix segv about skip_next, warn on unsupported cases of linking options (#1958)

* todos

* ensure afl-cc only allows available compiler modes

* update grammar mutator

* disable aslr on apple

* fix for arm64

* help selective instrumentation

* typos

* macos

* add compiler test script

* apple fixes

* bump nyx submodules (#1963)

* fix docs

* update changelog

* update grammar mutator

* improve compiler test script

* gcc asan workaround (#1966)

* fix github merge fuckup

* fix

* Fix afl-cc (#1968)

- Check if too many cmdline params here, each time before insert a new param.
 - Check if it is "-fsanitize=..." before we do sth.
 - Remove improper param_st transfer.

* Avoid adding llvmnative instrumentation when linking rust sanitizer runtime (#1969)

* Dynamic instrumentation filtering for LLVM native (#1971)

* Add two dynamic instrumentation filter methods to runtime

* Always use pc-table with native pcguard

* Add make_symbol_list.py and README

* changelog

* todos

* new forkserver check

* fix

* nyx test for CI

* improve nyx docs

* Fixes to afl-cc and documentation (#1974)

* Always compile with -ldl when building for CODE_COVERAGE

When building with CODE_COVERAGE, the afl runtime contains code that
calls `dladdr` which requires -ldl. Under most circumstances, clang
already adds this (e.g. when building with pc-table), but there are some
circumstances where it isn't added automatically.

* Add visibility declaration to __afl_connected

When building with hidden visibility, the use of __AFL_LOOP inside such
code can cause linker errors due to __afl_connected being declared
"hidden".

* Update docs to clarify that CODE_COVERAGE=1 is required for dynamic_covfilter

* nits

* nyx build script updates

* test error output

* debug ci

* debug ci

* Improve afl-cc (#1975)

* update response file support

 - full support of rsp file
 - fix some segv issues

* Improve afl-cc

 - remove dead code about allow/denylist options of sancov
 - missing `if (!aflcc->have_msan)`
 - add docs for each function
 - typo

* enable nyx

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* fix ci

* clean test script

* NO_NYX

* NO_NYX

* fix ci

* debug ci

* fix ci

* finalize ci fix

---------

Signed-off-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Sonic <50692172+SonicStark@users.noreply.github.com>
Co-authored-by: Xeonacid <h.dwwwwww@gmail.com>
Co-authored-by: Nils Bars <nils.bars@rub.de>
Co-authored-by: Jean-Romain Garnier <7504819+JRomainG@users.noreply.github.com>
Co-authored-by: Jean-Romain Garnier <jean-romain.garnier@airbus.com>
Co-authored-by: Sergej Schumilo <sergej@schumilo.de>
Co-authored-by: Christian Holler (:decoder) <choller@mozilla.com>

1 files changed, 68 insertions, 9 deletions

diff --git a/nyx_mode/build_nyx_support.sh b/nyx_mode/build_nyx_support.sh
index 581a8292..fda4ec12 100755
--- a/nyx_mode/build_nyx_support.sh
+++ b/nyx_mode/build_nyx_support.sh
@@ -9,6 +9,21 @@ echo
 
 echo "[*] Performing basic sanity checks..."
 
+if [ "$CI" = "true" ]; then
+
+  echo "[-] Error: nyx_mode cannot be tested in the Github CI, skipping ..."
+  exit 0
+
+fi
+
+
+if [ -n "$NO_NYX" ]; then
+
+  echo "[-] Error: the NO_NYX environment variable is set, please unset."
+  exit 0
+
+fi
+
 if [ ! "$(uname -s)" = "Linux" ]; then
 
   echo "[-] Error: Nyx mode is only available on Linux."
@@ -23,11 +38,17 @@ if [ ! "$(uname -m)" = "x86_64" ]; then
 
 fi
 
+cargo help > /dev/null 2>&1 || {
+   echo "[-] Error: Rust is not installed."
+   exit 0
+}
+
 echo "[*] Making sure all Nyx is checked out"
 
 
 if git status 1>/dev/null 2>&1; then
 
+  set +e
   git submodule init
   echo "[*] initializing QEMU-Nyx submodule"
   git submodule update ./QEMU-Nyx 2>/dev/null # ignore errors
@@ -35,6 +56,7 @@ if git status 1>/dev/null 2>&1; then
   git submodule update ./packer 2>/dev/null # ignore errors
   echo "[*] initializing libnyx submodule"
   git submodule update ./libnyx 2>/dev/null # ignore errors
+  set -e
 
 else
 
@@ -48,20 +70,57 @@ test -e packer/.git || { echo "[-] packer not checked out, please install git or
 test -e libnyx/.git || { echo "[-] libnyx not checked out, please install git or check your internet connection." ; exit 1 ; }
 test -e QEMU-Nyx/.git || { echo "[-] QEMU-Nyx not checked out, please install git or check your internet connection." ; exit 1 ; }
 
-echo "[*] checking packer init.cpio.gz ..."
-if [ ! -f "packer/linux_initramfs/init.cpio.gz" ]; then
-    (cd packer/linux_initramfs/ && sh pack.sh)
+
+QEMU_NYX_VERSION="$(cat ./QEMU_NYX_VERSION)"
+cd "./QEMU-Nyx" || exit 1
+if [ -n "$NO_CHECKOUT" ]; then
+  echo "[*] Skipping checkout to $QEMU_NYX_VERSION"
+else
+  echo "[*] Checking out $QEMU_NYX_VERSION"
+  set +e
+  sh -c 'git stash' 1>/dev/null 2>/dev/null
+  git pull 1>/dev/null 2>/dev/null
+  git checkout "$QEMU_NYX_VERSION" || echo Warning: could not check out to commit $QEMU_NYX_VERSION
+  set -e
 fi
+cd - > /dev/null
 
-echo "[*] Checking libnyx ..."
-if [ ! -f "libnyx/libnyx/target/release/liblibnyx.a" ]; then
-    (cd libnyx/libnyx && cargo build --release)
+PACKER_VERSION="$(cat ./PACKER_VERSION)"
+cd "./packer" || exit 1
+if [ -n "$NO_CHECKOUT" ]; then
+  echo "[*] Skipping checkout to $PACKER_VERSION"
+else
+  echo "[*] Checking out $PACKER_VERSION"
+  set +e
+  sh -c 'git stash' 1>/dev/null 2>/dev/null
+  git pull 1>/dev/null 2>/dev/null
+  git checkout "$PACKER_VERSION" || echo Warning: could not check out to commit $PACKER_VERSION
+  set -e
 fi
+cd - > /dev/null
 
-echo "[*] Checking QEMU-Nyx ..."
-if [ ! -f "QEMU-Nyx/x86_64-softmmu/qemu-system-x86_64" ]; then
-    (cd QEMU-Nyx && ./compile_qemu_nyx.sh static)
+LIBNYX_VERSION="$(cat ./LIBNYX_VERSION)"
+cd "./libnyx/" || exit 1
+if [ -n "$NO_CHECKOUT" ]; then
+  echo "[*] Skipping checkout to $LIBNYX_VERSION"
+else
+  echo "[*] Checking out $LIBNYX_VERSION"
+  set +e
+  sh -c 'git stash' 1>/dev/null 2>/dev/null
+  git pull 1>/dev/null 2>/dev/null
+  git checkout "$LIBNYX_VERSION" || echo Warning: could not check out to commit $LIBNYX_VERSION
+  set -e
 fi
+cd - > /dev/null
+
+echo "[*] checking packer init.cpio.gz ..."
+(cd packer/linux_initramfs/ && sh pack.sh)
+
+echo "[*] Checking libnyx ..."
+(cd libnyx/libnyx && cargo build --release)
+
+echo "[*] Checking QEMU-Nyx ..."
+(cd QEMU-Nyx && ./compile_qemu_nyx.sh static )
 
 echo "[*] Checking libnyx.so ..."
 cp libnyx/libnyx/target/release/liblibnyx.so ../libnyx.so