1 00:00:00,120 --> 00:00:02,900 As I mentioned at the beginning of the class, there are many, many, 2 00:00:02,900 --> 00:00:06,150 many, white box criteria. And we're not going to have time to cover them 3 00:00:06,150 --> 00:00:08,000 all. So what I like to do now is just to give you 4 00:00:08,000 --> 00:00:11,290 the flavor, of some other criteria by just mentioning them, and saying a 5 00:00:11,290 --> 00:00:13,740 couple of things, about the way they work. And the first one I 6 00:00:13,740 --> 00:00:15,370 want to mention is path coverage. And 7 00:00:15,370 --> 00:00:17,800 in path coverage, the test requirements are 8 00:00:17,800 --> 00:00:21,220 all the paths in the program. So what that means is that to 9 00:00:21,220 --> 00:00:25,250 satisfy this criteria, we need to generate enough test cases such that all 10 00:00:25,250 --> 00:00:27,230 the paths in my program are covered. As you can 11 00:00:27,230 --> 00:00:31,970 imagine this is incredibly expensive because any nontrivial program has got 12 00:00:31,970 --> 00:00:34,280 a virtual infinite number of paths and therefore we will 13 00:00:34,280 --> 00:00:36,910 need a virtual infinite number of test cases to satisfy the 14 00:00:36,910 --> 00:00:39,950 path coverage criteria. Another family of criteria I want to mention 15 00:00:39,950 --> 00:00:43,970 are data-flow coverage criteria and in data-flow coverage criteria, the focus 16 00:00:43,970 --> 00:00:47,480 shifts from the coverage of individual elements in the code to 17 00:00:47,480 --> 00:00:50,320 the coverage of pairs of elements. And in particular the coverage 18 00:00:50,320 --> 00:00:53,770 of Statements, in which the content of some memory locations 19 00:00:53,770 --> 00:00:56,920 modified, and statements in which the content of the same 20 00:00:56,920 --> 00:00:59,860 memory location is used. So in this way, our test 21 00:00:59,860 --> 00:01:03,030 will exercise the assignments of values to memory, and the 22 00:01:03,030 --> 00:01:06,020 usage of those assignments,. Finally, I want to mention mutation 23 00:01:06,020 --> 00:01:09,150 coverage. And this is a fairly different and new idea, 24 00:01:09,150 --> 00:01:11,910 so the key concept in mutation coverage is that we 25 00:01:11,910 --> 00:01:16,156 want to evaluate the goodness of our test by modifying the code. 26 00:01:16,156 --> 00:01:18,704 For example here in this small I'm, I'm showing that I might 27 00:01:18,704 --> 00:01:21,820 change it. An if statement from K greater than 9, to K 28 00:01:21,820 --> 00:01:24,310 greater than or equal to 9. And the reason why I want to 29 00:01:24,310 --> 00:01:27,750 do that, is that if I generate enough mutants, enough variation of 30 00:01:27,750 --> 00:01:30,340 my program, then I can use them to assess how good are 31 00:01:30,340 --> 00:01:34,330 my tests at distinguishing the original program and the mutants. And because 32 00:01:34,330 --> 00:01:37,430 I'm changing the code based on the way I expect to introduce 33 00:01:37,430 --> 00:01:41,370 errors in the code, the more my test can identify mutants, the better 34 00:01:41,370 --> 00:01:44,140 they are at identifying real faults. This is a very high 35 00:01:44,140 --> 00:01:46,470 level [UNKNOWN], but just to give you the intuition and the idea 36 00:01:46,470 --> 00:01:50,210 behind these criteria, and I'm going to provide as usual, additional pointers 37 00:01:50,210 --> 00:01:52,930 to go in more depth on these topics in the notes of 38 00:01:52,930 --> 00:01:55,170 the class. So now, let's go back to our test criteria 39 00:01:55,170 --> 00:01:59,310 subsumption hierarchy, and see how all of these criteria fit together in 40 00:01:59,310 --> 00:02:02,663 this context. Let me start with multiple condition coverage. As we saw, 41 00:02:02,663 --> 00:02:06,740 MC/DC is sort of as [UNKNOWN] way of doing multiple condition coverage 42 00:02:06,740 --> 00:02:09,229 in the sense that it doesn't consider all the combinations, 43 00:02:09,229 --> 00:02:11,460 but only the ones that are more likely to matter. 44 00:02:11,460 --> 00:02:14,930 And as such, MC/DC exercises a subset of the elements 45 00:02:14,930 --> 00:02:16,220 of the multiple condition coverage 46 00:02:16,220 --> 00:02:18,760 exercises. And therefore, multiple condition coverage 47 00:02:18,760 --> 00:02:22,060 is more thorough than MC/DC and subsumption, and it's also 48 00:02:22,060 --> 00:02:26,300 as we saw incredibly more expensive. Path coverage subsumes branch coverage, 49 00:02:26,300 --> 00:02:28,090 because if I cover all of the paths in the 50 00:02:28,090 --> 00:02:32,140 code, I necessarily cover all the branches. However, it doesn't subsume 51 00:02:32,140 --> 00:02:34,130 multiple condition coverage, MC/CD, or 52 00:02:34,130 --> 00:02:35,910 branch and condition coverage. Because this 53 00:02:35,910 --> 00:02:38,590 criteria, have additional requirements involving 54 00:02:38,590 --> 00:02:39,940 the conditions of the predicate that 55 00:02:39,940 --> 00:02:42,730 path coverage does not have. As for data-flow coverage criteria and 56 00:02:42,730 --> 00:02:45,010 mutation coverage criteria, there really no 57 00:02:45,010 --> 00:02:46,860 relation with the other criteria, because 58 00:02:46,860 --> 00:02:49,710 they look at different aspects, of the code. So they're not 59 00:02:49,710 --> 00:02:52,220 really comparable, and therefore we're just going to put them on the 60 00:02:52,220 --> 00:02:55,000 side, without any relationship with the other ones. The reason why 61 00:02:55,000 --> 00:02:57,300 I want to represent them all anyways is because I want to make 62 00:02:57,300 --> 00:02:59,950 an important distinction between these different criteria. And 63 00:02:59,950 --> 00:03:02,690 that's the distinction between practical criteria, which are 64 00:03:02,690 --> 00:03:05,490 criteria that are actually not too expansive to 65 00:03:05,490 --> 00:03:08,790 be used in real scenarios. And theoretical criteria, which 66 00:03:08,790 --> 00:03:11,590 are criteria that are useful in theory from 67 00:03:11,590 --> 00:03:14,140 the conceptual standpoint, but they're not really applicable 68 00:03:14,140 --> 00:03:16,540 in practice because they're too expansive, because they 69 00:03:16,540 --> 00:03:19,460 require basically too many test cases to be satisfied.