Fix saturated maps & stability cliff in recalibration

I have observed two problems:

  1. A sudden "stability cliff" where stability drops precipitously.

  2. A sudden jump to a 100% saturated "density map".

Both issues are due to attempted "recalibration" of a case at the
beginning of fuzz_one_original() or mopt_common_fuzzing().  See the
comments "CALIBRATION (only if failed earlier on)" in those functions
and the subsequent call to calibrate_case().

At those calls to calibrate_case(), afl->fsrv.trace_bits holds
trace_bits for a run of the SUT on a prior queue entry.  However,
calibrate_case() may use the trace_bits as if they apply to the
current queue entry (afl->queue_cur).

Most often this bug causes the "stability cliff".  Trace bits are
compared for runs on distinct inputs, which can be very different.
The result is a sudden drop in stability.

Sometimes it leads to the "saturated map" problem.  A saturated
density map arises if the trace bits on the previous entry were
"simplified" by simplify_trace().  Simplified traces only contain the
values 1 and 128.  They are meant to be compared against
virgin_crashes and virgin_tmouts.

However, this bug causes the (stale) simplified trace to be compared
against virgin_bits during a call to has_new_bits(), which causes
every byte in vigin_bits to be something other than 255.  The overall
map density is determined by the percentage of bytes not 255, which
will be 100%.  Worse, AFL++ will be unable to detect novel occurrences
of edge counts 1 and 128 going forward.

This patch avoids the above issues by clearing q->exec_cksum when
calibration fails.  Recalibrations are forced to start with a fresh
trace on the queue entry.

Thanks to @andreafioraldi for suggesting the current, improved patch.

0 files changed, 0 insertions, 0 deletions