1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
|
1
00:00:00,400 --> 00:00:03,780
So now that we know what the CFG is, let's see how we can get into a count,
2
00:00:03,780 --> 00:00:07,860
that else part, that is missing in our example code by introducing a new
3
00:00:07,860 --> 00:00:10,480
kind of coverage. Branch coverage. As usual,
4
00:00:10,480 --> 00:00:13,550
I'm going to describe branch coverage in terms of test requirements and
5
00:00:13,550 --> 00:00:17,100
coverage measure. So starting from test requirements. The test requirement for
6
00:00:17,100 --> 00:00:21,080
branch coverage are the branches in the program. In other words,
7
00:00:21,080 --> 00:00:25,240
the goal of branch coverage is to execute all of the branches in the program.
8
00:00:25,240 --> 00:00:28,370
The coverage measure is defined accordingly as the number of branches
9
00:00:28,370 --> 00:00:33,150
executed by my test cases over the total number of branches in the program. And
10
00:00:33,150 --> 00:00:37,020
let me remind you that branches are the outgoing edges from a decision point.
11
00:00:37,020 --> 00:00:40,410
Therefore, an if statement, a switch statement, a while statement.
12
00:00:40,410 --> 00:00:44,400
Any note in the c of g that has got more than one outgoing edge.
13
00:00:44,400 --> 00:00:48,170
Those edges are called branches. So let's look at that using our example. So
14
00:00:48,170 --> 00:00:51,690
now we're looking back at our printSum example. And in addition to the code,
15
00:00:51,690 --> 00:00:54,440
I also want to represent the CFG for the program. So
16
00:00:54,440 --> 00:00:58,590
let's start by looking at how many branches we have in our code. Which means how
17
00:00:58,590 --> 00:01:02,070
many test requirements we have. And in this case there are two decision points.
18
00:01:02,070 --> 00:01:04,940
The first one that corresponds to the first if, and the second one that
19
00:01:04,940 --> 00:01:10,100
corresponds to the second if. So we have one, two, three, and four branches. So
20
00:01:10,100 --> 00:01:13,880
now, let's bring back our current set of test cases. We had two test cases.
21
00:01:13,880 --> 00:01:17,440
The one's that, with which we achieved a 100% statement coverage. And
22
00:01:17,440 --> 00:01:21,140
let's see what happens in terms of branch coverage when we run these test cases.
23
00:01:21,140 --> 00:01:24,110
I start from the first one, when we execute it, we follow the code,
24
00:01:24,110 --> 00:01:27,890
we get to this decision point because the predicate in the if statement is true.
25
00:01:27,890 --> 00:01:30,770
We follow the true branch, therefore we get here and then,
26
00:01:30,770 --> 00:01:34,970
we exit from the program. So, in this case, we covered one of the branches,
27
00:01:34,970 --> 00:01:40,710
which means that we got to 25% coverage. Now when we run the second test case,
28
00:01:40,710 --> 00:01:44,160
again we follow this path. We get to this, the first if and
29
00:01:44,160 --> 00:01:47,730
in this case the predicate of the if is false. Therefore, we go this way.
30
00:01:47,730 --> 00:01:51,160
We reach the second predicate, the second if. The result is true, so
31
00:01:51,160 --> 00:01:55,490
we follow the true branch and therefore, we cover these additional two branches.
32
00:01:55,490 --> 00:01:59,900
So at this point, we are at 75% branch coverage. So what
33
00:01:59,900 --> 00:02:04,070
happens is that we're missing this branch. For now, the inputs that we consider,
34
00:02:04,070 --> 00:02:07,740
this branch is executed. Therefore, we need to add an additional test case. And
35
00:02:07,740 --> 00:02:11,600
that this case that we need, is one for which this predicate is false and this
36
00:02:11,600 --> 00:02:15,680
predicate is false. The simplest possibility in this case is the test case for
37
00:02:15,680 --> 00:02:20,170
which A is equal to 0 and B is equal to 0. If we execute this test case,
38
00:02:20,170 --> 00:02:24,250
our execution again followed this path, follows the fourth branch here.
39
00:02:24,250 --> 00:02:28,090
And in this case, because result is not less than zero either, will follow
40
00:02:28,090 --> 00:02:33,090
this branch as well. And therefore, we will reach our 100% branch coverage. And
41
00:02:33,090 --> 00:02:35,820
this covered the problem. Something that I would like to clarify before we
42
00:02:35,820 --> 00:02:40,920
move to the next topic, is that 100% coverage does not provide any guarantee of
43
00:02:40,920 --> 00:02:44,530
finding the problems in the code. All we saw so far is the fact that by
44
00:02:44,530 --> 00:02:48,580
testing more thoroughly we have more chances of finding a problem in the code.
45
00:02:48,580 --> 00:02:51,680
But it doesn't matter which kind of coverage we utilize, and how much
46
00:02:51,680 --> 00:02:55,440
coverage we achieve. There's always a chance that we might miss something. And
47
00:02:55,440 --> 00:02:59,760
I will get back to this later on in the lesson. I just mentioned the fact that
48
00:02:59,760 --> 00:03:03,910
we tested more fully when we went from statement coverage to branch coverage.
49
00:03:03,910 --> 00:03:06,900
What does that mean exactly? To explain that, I'm going to introduce the concept
50
00:03:06,900 --> 00:03:11,420
of test criteria subsumption. One test criteria subsumes another criteria when
51
00:03:11,420 --> 00:03:16,760
all the tests widths that satisfy that criteria will also satisfy the other one.
52
00:03:16,760 --> 00:03:18,740
So let me show you that with statement and
53
00:03:18,740 --> 00:03:23,220
branch coverage. If we identify a test width that achieves 100% branch coverage,
54
00:03:23,220 --> 00:03:27,780
the same test width will also achieve, necessarily, 100% statement coverage.
55
00:03:27,780 --> 00:03:31,290
That's what happened for our example, and also what happens in general,
56
00:03:31,290 --> 00:03:35,660
because branch coverage is a stronger criteria than statement coverage.
57
00:03:35,660 --> 00:03:39,670
There is no way to cover all the branches without covering all the statements.
58
00:03:39,670 --> 00:03:43,480
It is not true that any test results satisfies statement coverage will also
59
00:03:43,480 --> 00:03:47,040
satisfy branch coverage. And, in fact, we just saw a counter example.
60
00:03:47,040 --> 00:03:50,569
When we look at the printSum code. We had a test where there was achieving 100%
61
00:03:50,569 --> 00:03:53,910
statement coverage and was not achieving 100% branch coverage. Therefore,
62
00:03:53,910 --> 00:03:58,301
in this case we have a substantial relation in this direction. Branch coverage,
63
00:03:58,301 --> 00:04:02,860
subsumes statement coverage. What it also means is that normally, or in general,
64
00:04:02,860 --> 00:04:06,910
it is more expensive to achieve branch coverage than achieve statement coverage,
65
00:04:06,910 --> 00:04:09,490
because achieving branch coverage requires the generation of
66
00:04:09,490 --> 00:04:13,770
a larger number of test cases. So what this relation means is that
67
00:04:13,770 --> 00:04:17,779
branch coverage is stronger than statement coverage but also more expensive.
|
