1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
|
1
00:00:00,120 --> 00:00:02,900
As I mentioned at the beginning of the class, there are many, many,
2
00:00:02,900 --> 00:00:06,150
many, white box criteria. And we're not going to have time to cover them
3
00:00:06,150 --> 00:00:08,000
all. So what I like to do now is just to give you
4
00:00:08,000 --> 00:00:11,290
the flavor, of some other criteria by just mentioning them, and saying a
5
00:00:11,290 --> 00:00:13,740
couple of things, about the way they work. And the first one I
6
00:00:13,740 --> 00:00:15,370
want to mention is path coverage. And
7
00:00:15,370 --> 00:00:17,800
in path coverage, the test requirements are
8
00:00:17,800 --> 00:00:21,220
all the paths in the program. So what that means is that to
9
00:00:21,220 --> 00:00:25,250
satisfy this criteria, we need to generate enough test cases such that all
10
00:00:25,250 --> 00:00:27,230
the paths in my program are covered. As you can
11
00:00:27,230 --> 00:00:31,970
imagine this is incredibly expensive because any nontrivial program has got
12
00:00:31,970 --> 00:00:34,280
a virtual infinite number of paths and therefore we will
13
00:00:34,280 --> 00:00:36,910
need a virtual infinite number of test cases to satisfy the
14
00:00:36,910 --> 00:00:39,950
path coverage criteria. Another family of criteria I want to mention
15
00:00:39,950 --> 00:00:43,970
are data-flow coverage criteria and in data-flow coverage criteria, the focus
16
00:00:43,970 --> 00:00:47,480
shifts from the coverage of individual elements in the code to
17
00:00:47,480 --> 00:00:50,320
the coverage of pairs of elements. And in particular the coverage
18
00:00:50,320 --> 00:00:53,770
of Statements, in which the content of some memory locations
19
00:00:53,770 --> 00:00:56,920
modified, and statements in which the content of the same
20
00:00:56,920 --> 00:00:59,860
memory location is used. So in this way, our test
21
00:00:59,860 --> 00:01:03,030
will exercise the assignments of values to memory, and the
22
00:01:03,030 --> 00:01:06,020
usage of those assignments,. Finally, I want to mention mutation
23
00:01:06,020 --> 00:01:09,150
coverage. And this is a fairly different and new idea,
24
00:01:09,150 --> 00:01:11,910
so the key concept in mutation coverage is that we
25
00:01:11,910 --> 00:01:16,156
want to evaluate the goodness of our test by modifying the code.
26
00:01:16,156 --> 00:01:18,704
For example here in this small I'm, I'm showing that I might
27
00:01:18,704 --> 00:01:21,820
change it. An if statement from K greater than 9, to K
28
00:01:21,820 --> 00:01:24,310
greater than or equal to 9. And the reason why I want to
29
00:01:24,310 --> 00:01:27,750
do that, is that if I generate enough mutants, enough variation of
30
00:01:27,750 --> 00:01:30,340
my program, then I can use them to assess how good are
31
00:01:30,340 --> 00:01:34,330
my tests at distinguishing the original program and the mutants. And because
32
00:01:34,330 --> 00:01:37,430
I'm changing the code based on the way I expect to introduce
33
00:01:37,430 --> 00:01:41,370
errors in the code, the more my test can identify mutants, the better
34
00:01:41,370 --> 00:01:44,140
they are at identifying real faults. This is a very high
35
00:01:44,140 --> 00:01:46,470
level [UNKNOWN], but just to give you the intuition and the idea
36
00:01:46,470 --> 00:01:50,210
behind these criteria, and I'm going to provide as usual, additional pointers
37
00:01:50,210 --> 00:01:52,930
to go in more depth on these topics in the notes of
38
00:01:52,930 --> 00:01:55,170
the class. So now, let's go back to our test criteria
39
00:01:55,170 --> 00:01:59,310
subsumption hierarchy, and see how all of these criteria fit together in
40
00:01:59,310 --> 00:02:02,663
this context. Let me start with multiple condition coverage. As we saw,
41
00:02:02,663 --> 00:02:06,740
MC/DC is sort of as [UNKNOWN] way of doing multiple condition coverage
42
00:02:06,740 --> 00:02:09,229
in the sense that it doesn't consider all the combinations,
43
00:02:09,229 --> 00:02:11,460
but only the ones that are more likely to matter.
44
00:02:11,460 --> 00:02:14,930
And as such, MC/DC exercises a subset of the elements
45
00:02:14,930 --> 00:02:16,220
of the multiple condition coverage
46
00:02:16,220 --> 00:02:18,760
exercises. And therefore, multiple condition coverage
47
00:02:18,760 --> 00:02:22,060
is more thorough than MC/DC and subsumption, and it's also
48
00:02:22,060 --> 00:02:26,300
as we saw incredibly more expensive. Path coverage subsumes branch coverage,
49
00:02:26,300 --> 00:02:28,090
because if I cover all of the paths in the
50
00:02:28,090 --> 00:02:32,140
code, I necessarily cover all the branches. However, it doesn't subsume
51
00:02:32,140 --> 00:02:34,130
multiple condition coverage, MC/CD, or
52
00:02:34,130 --> 00:02:35,910
branch and condition coverage. Because this
53
00:02:35,910 --> 00:02:38,590
criteria, have additional requirements involving
54
00:02:38,590 --> 00:02:39,940
the conditions of the predicate that
55
00:02:39,940 --> 00:02:42,730
path coverage does not have. As for data-flow coverage criteria and
56
00:02:42,730 --> 00:02:45,010
mutation coverage criteria, there really no
57
00:02:45,010 --> 00:02:46,860
relation with the other criteria, because
58
00:02:46,860 --> 00:02:49,710
they look at different aspects, of the code. So they're not
59
00:02:49,710 --> 00:02:52,220
really comparable, and therefore we're just going to put them on the
60
00:02:52,220 --> 00:02:55,000
side, without any relationship with the other ones. The reason why
61
00:02:55,000 --> 00:02:57,300
I want to represent them all anyways is because I want to make
62
00:02:57,300 --> 00:02:59,950
an important distinction between these different criteria. And
63
00:02:59,950 --> 00:03:02,690
that's the distinction between practical criteria, which are
64
00:03:02,690 --> 00:03:05,490
criteria that are actually not too expansive to
65
00:03:05,490 --> 00:03:08,790
be used in real scenarios. And theoretical criteria, which
66
00:03:08,790 --> 00:03:11,590
are criteria that are useful in theory from
67
00:03:11,590 --> 00:03:14,140
the conceptual standpoint, but they're not really applicable
68
00:03:14,140 --> 00:03:16,540
in practice because they're too expansive, because they
69
00:03:16,540 --> 00:03:19,460
require basically too many test cases to be satisfied.
|
