path: root/usth/ICT2.7/P4L2 Black-Box Testing Subtitles/23 - Category Partition Demo - lang_en_vs4.srt

1
00:00:01,290 --> 00:00:03,210
In this demo, we're going to do exactly what we

2
00:00:03,210 --> 00:00:06,370
did just now in the lesson. We're going to use

3
00:00:06,370 --> 00:00:09,250
the category partition method to go from a high-level

4
00:00:09,250 --> 00:00:12,930
description of a piece of software of a program to

5
00:00:12,930 --> 00:00:15,512
a set of test cases for that program. To

6
00:00:15,512 --> 00:00:17,640
do that, we're going to use a simple tool. So

7
00:00:17,640 --> 00:00:21,040
I'm going to show you here the tool that is

8
00:00:21,040 --> 00:00:26,380
called a tsl generator right here. This tool is available

9
00:00:26,380 --> 00:00:28,520
to you, so you can look in the class notes to

10
00:00:28,520 --> 00:00:31,900
see information on how to download it. And together with the tool,

11
00:00:31,900 --> 00:00:34,890
we are also going to provide a manual for the tool, and

12
00:00:34,890 --> 00:00:37,680
a set of files that I'm going to use in this demo. So

13
00:00:37,680 --> 00:00:40,420
you should be able to do exactly what I'm doing. So

14
00:00:40,420 --> 00:00:45,110
again, all of those are available from the class notes. So specifically,

15
00:00:45,110 --> 00:00:48,390
today we're going to write test cases for the grep program. So in

16
00:00:48,390 --> 00:00:51,970
case you're familiar with the grep utility, this is a simplified version

17
00:00:51,970 --> 00:00:55,060
of that utility. So basically the grep utility allows you

18
00:00:55,060 --> 00:00:58,410
to search a file for the occurrences of a given

19
00:00:58,410 --> 00:01:01,552
pattern. So you can invoke it, as it's shown here

20
00:01:01,552 --> 00:01:05,570
in the synopsis, by executing grep, the pattern that you're

21
00:01:05,570 --> 00:01:08,310
looking for, and the filename in which you want to

22
00:01:08,310 --> 00:01:10,300
look for the pattern. And let me read the description

23
00:01:10,300 --> 00:01:13,650
of the grep utility. The grep utility searches files for

24
00:01:13,650 --> 00:01:17,110
a pattern and brings all lines that contain that pattern

25
00:01:17,110 --> 00:01:20,900
on the standard output. A line that contains multiple occurrences of

26
00:01:20,900 --> 00:01:24,350
the pattern is printed only once. The pattern is any sequence

27
00:01:24,350 --> 00:01:27,700
of characters. To include a blank in the pattern, the entire

28
00:01:27,700 --> 00:01:31,060
pattern must be enclosed in single quotes. To include a quote

29
00:01:31,060 --> 00:01:34,420
sign in the pattern, the quote sign must be escaped, which

30
00:01:34,420 --> 00:01:36,260
means that we have to put a slash in front of

31
00:01:36,260 --> 00:01:39,290
the quotes sign. And in general, it is safest to enclose

32
00:01:39,290 --> 00:01:42,210
the entire pattern in single quotes. So this is our high

33
00:01:42,210 --> 00:01:45,420
level description for the program, for the softer system, that

34
00:01:45,420 --> 00:01:47,270
we need to test. So now let me show you

35
00:01:47,270 --> 00:01:50,600
what a possible set of categories and partitions could be

36
00:01:50,600 --> 00:01:53,770
for this program. So what I have here is a

37
00:01:53,770 --> 00:01:58,080
file, a textual file, which contains all the categories and

38
00:01:58,080 --> 00:02:02,760
partitions for the elements that are relevant for my program.

39
00:02:02,760 --> 00:02:05,240
In particular, when we look at the file, we can

40
00:02:05,240 --> 00:02:07,809
see that the file can be characterized by its size.

41
00:02:08,889 --> 00:02:12,160
And in this case, I've got two choices. The file can

42
00:02:12,160 --> 00:02:16,050
be empty or not empty. The second characteristic of the file

43
00:02:16,050 --> 00:02:19,490
that I'm considering is the number of occurrences of the pattern

44
00:02:19,490 --> 00:02:22,320
in the file. And I'm considering that the pattern might not occur

45
00:02:22,320 --> 00:02:25,780
in the file or it might occur once, or multiple times.

46
00:02:25,780 --> 00:02:28,264
I'm not going to go through the rest of the file because we

47
00:02:28,264 --> 00:02:31,234
already covered how to apply the category partition method in the

48
00:02:31,234 --> 00:02:34,226
lesson. So if you had doubts about that, about the method and

49
00:02:34,226 --> 00:02:36,952
how to apply, you might want to go back and watch again the

50
00:02:36,952 --> 00:02:40,040
lesson. What I want to show you here is how you can go

51
00:02:40,040 --> 00:02:43,670
from this information that you have here, that we have derived by

52
00:02:43,670 --> 00:02:47,020
applying the, the first steps of the method, to a set of

53
00:02:47,020 --> 00:02:50,650
test frames, and then, a set of test packs. So to do

54
00:02:50,650 --> 00:02:53,240
that we're going to use the tool that I just mentioned. So let

55
00:02:53,240 --> 00:02:56,670
me bring back my terminal. So first of all, let's see how

56
00:02:56,670 --> 00:02:59,570
we can run the tool. So you have a manual that will explain

57
00:02:59,570 --> 00:03:02,180
all the details on how to build the file that we're

58
00:03:02,180 --> 00:03:04,350
going to feed the tool. So what is the format and so

59
00:03:04,350 --> 00:03:07,780
on. Here I'm just going to see how I can run the

60
00:03:07,780 --> 00:03:10,828
tool. So first of all, let me point out that this was

61
00:03:10,828 --> 00:03:15,028
developed together by professors from the University of California Irvine and

62
00:03:15,028 --> 00:03:18,020
Oregon State University. And as you can see, we can run

63
00:03:18,020 --> 00:03:20,968
TSL generator and specify that we want to see the main

64
00:03:20,968 --> 00:03:24,361
page. So in this case if we run it this, this way,

65
00:03:24,361 --> 00:03:27,723
you'll have some basic information on how to run

66
00:03:27,723 --> 00:03:30,720
the tool. And from the main page you can see

67
00:03:30,720 --> 00:03:33,520
that you can specify the minus c flag and in

68
00:03:33,520 --> 00:03:37,360
this case the TSL generator will report the number of

69
00:03:37,360 --> 00:03:41,410
test frames generated without writing them to output. For

70
00:03:41,410 --> 00:03:43,828
example, you might want to use this as we will do

71
00:03:43,828 --> 00:03:46,308
to see how many tests that you will generate with

72
00:03:46,308 --> 00:03:49,630
a current set of category partitions and choices. The minus

73
00:03:49,630 --> 00:03:52,620
s option will bring the result of the TSL

74
00:03:52,620 --> 00:03:55,620
generator on the standard output. And finally, you can

75
00:03:55,620 --> 00:03:58,330
use minus o to specify an output file, where

76
00:03:58,330 --> 00:04:01,010
to put the output of the program. So let's

77
00:04:01,010 --> 00:04:05,070
at first run our TSL generator by specifying the

78
00:04:05,070 --> 00:04:08,620
minus c option and by bypassing our current set

79
00:04:08,620 --> 00:04:11,860
of category partitions and choices. Okay, so let me

80
00:04:11,860 --> 00:04:15,140
remind you that what the, the tool will do

81
00:04:15,140 --> 00:04:17,790
is what we will do manually. Otherwise, which is

82
00:04:17,790 --> 00:04:20,380
to combine all these choices so as to have one

83
00:04:20,380 --> 00:04:23,305
test case for each combination. So if we do

84
00:04:23,305 --> 00:04:27,412
that, you can see that the tool tells us that

85
00:04:27,412 --> 00:04:32,383
we will generate 7776 test frames in this case.

86
00:04:32,383 --> 00:04:34,660
And this seems to be a little too much for

87
00:04:34,660 --> 00:04:36,868
a program as small as the one that we are

88
00:04:36,868 --> 00:04:40,341
testing. And assume for instance that we don't have the

89
00:04:40,341 --> 00:04:43,149
resources to run this many test cases for, for

90
00:04:43,149 --> 00:04:46,518
the grep program. In addition, consider that in this case,

91
00:04:46,518 --> 00:04:50,356
we're computing all possible combinations of choices. And there's going to

92
00:04:50,356 --> 00:04:52,384
be some combination that do not make sense as we

93
00:04:52,384 --> 00:04:54,945
discussed in the lesson. So what we might want to

94
00:04:54,945 --> 00:04:57,051
do in this case is to go back to our

95
00:04:57,051 --> 00:05:03,120
spec and start adding constraints to eliminate this meaningless combination.

96
00:05:03,120 --> 00:05:05,980
So I'm going to show you the result of doing that.

97
00:05:05,980 --> 00:05:08,690
And I'm going to show you a few examples.

98
00:05:08,690 --> 00:05:11,670
For example here, when the file is empty, I'm going to

99
00:05:11,670 --> 00:05:15,010
define this property empty file. And how am I

100
00:05:15,010 --> 00:05:18,490
going to use this property? Well for example here, it

101
00:05:18,490 --> 00:05:20,760
doesn't make sense to consider the case in which

102
00:05:20,760 --> 00:05:24,660
we have one or many occurrences of the pattern in

103
00:05:24,660 --> 00:05:27,020
the file if the file is empty. Therefore I'm

104
00:05:27,020 --> 00:05:31,650
going to tell the tool that it should consider this specific

105
00:05:31,650 --> 00:05:35,780
choice only if the file is not empty, only if

106
00:05:35,780 --> 00:05:38,660
empty file is not defined. And that will skip, for

107
00:05:38,660 --> 00:05:41,330
example, all of the combinations in which the file is

108
00:05:41,330 --> 00:05:44,171
empty. And I'm trying to generate the test case that has

109
00:05:44,171 --> 00:05:46,364
one occurrence of the pattern in the file, which is

110
00:05:46,364 --> 00:05:49,355
simply not possible. For another example, in case I have

111
00:05:49,355 --> 00:05:52,824
an empty pattern, I define the property empty pattern. And

112
00:05:52,824 --> 00:05:56,725
then I avoid the choices that involve the pattern in case

113
00:05:56,725 --> 00:05:59,820
the pattern is empty. because, for example, I cannot

114
00:05:59,820 --> 00:06:03,900
have quotes in a pattern that is empty. For example,

115
00:06:03,900 --> 00:06:06,760
it doesn't make sense to have blanks. So, one or

116
00:06:06,760 --> 00:06:10,250
more blanks if the pattern is empty. So I'm going to

117
00:06:10,250 --> 00:06:14,180
specify again that this choice should be considered only if

118
00:06:14,180 --> 00:06:16,140
we don't have an empty pattern. And so on and

119
00:06:16,140 --> 00:06:20,080
so forth. So now after I edit these constraints, I

120
00:06:20,080 --> 00:06:21,890
can go back and compute again the number of test

121
00:06:21,890 --> 00:06:23,970
frames and therefore the test cases that will be

122
00:06:23,970 --> 00:06:27,530
generated with these constraints. So let me go again

123
00:06:27,530 --> 00:06:30,381
to my terminal. Okay, so now I'm going to run

124
00:06:30,381 --> 00:06:34,061
my TSL generator again, and I'm going to run it on

125
00:06:34,061 --> 00:06:37,389
the second version of this file. And you can

126
00:06:37,389 --> 00:06:40,546
see that I reduced the, the number of test frames

127
00:06:40,546 --> 00:06:43,889
from about 7800 to about 1700. So it's quite

128
00:06:43,889 --> 00:06:46,967
a, quite a big reduction by eliminating all these combinations

129
00:06:46,967 --> 00:06:49,540
that do not make sense. But let's assume again that

130
00:06:49,540 --> 00:06:52,040
we want to reduce this further so that we don't want to

131
00:06:52,040 --> 00:06:55,610
generate those many test frames and therefore test cases. So

132
00:06:55,610 --> 00:06:58,660
what can we do? We go back to our spec. And

133
00:06:58,660 --> 00:07:02,280
in this case, we start adding error constraints. So if

134
00:07:02,280 --> 00:07:05,200
you remember what we said in the lesson, error constraints are

135
00:07:05,200 --> 00:07:08,310
constraints that indicate a choice that has to do with an

136
00:07:08,310 --> 00:07:11,980
erroneous behaviour. For example, an erroneous input provided to the problem.

137
00:07:11,980 --> 00:07:15,210
So here for instance, we're indicating the presence

138
00:07:15,210 --> 00:07:20,060
of incorrectly enclosing quotes as an error choice. Same

139
00:07:20,060 --> 00:07:22,270
thing if there's no file corresponding to the

140
00:07:22,270 --> 00:07:23,970
name that we provide to the tool, we say

141
00:07:23,970 --> 00:07:26,760
that this corresponds to an error. So how

142
00:07:26,760 --> 00:07:29,130
is the tool going to use this information? It uses

143
00:07:29,130 --> 00:07:33,980
this information by producing only one combination that involves

144
00:07:33,980 --> 00:07:37,270
error choices, instead of combining them with other choices.

145
00:07:37,270 --> 00:07:39,780
So let's see what happens after we added this

146
00:07:39,780 --> 00:07:43,370
error constraints. So we go back to our console

147
00:07:43,370 --> 00:07:46,920
once more. And in this case, we want to run

148
00:07:46,920 --> 00:07:50,910
the TSL generator with the version of the, of my

149
00:07:50,910 --> 00:07:53,900
file that contains the area of constraints. And again,

150
00:07:53,900 --> 00:07:56,390
I reduce quite a bit the number of test frames.

151
00:07:56,390 --> 00:07:59,110
So now I have only 562 test frames that

152
00:07:59,110 --> 00:08:02,660
will be generated by using the file that I provided.

153
00:08:02,660 --> 00:08:05,460
So for the last time, let's assume that we really want

154
00:08:05,460 --> 00:08:07,780
to cut down the number of test frames or the number of

155
00:08:07,780 --> 00:08:10,380
test cases. So once more, we go back to our file, and

156
00:08:10,380 --> 00:08:12,980
at this point what we can add is the final type of

157
00:08:12,980 --> 00:08:14,170
constraints that we have, which are

158
00:08:14,170 --> 00:08:17,245
single constraints. And single constraints are

159
00:08:17,245 --> 00:08:21,360
basically indicated choices that we don't want to combine with other choices.

160
00:08:21,360 --> 00:08:24,210
So they have the same effect of the error constraints, but they

161
00:08:24,210 --> 00:08:28,120
have a different meaning, so they do not indicate choices that corresponds

162
00:08:28,120 --> 00:08:29,860
to an error. In other words, I can use a

163
00:08:29,860 --> 00:08:35,280
single constraints to identify choices that I want to test only once.

164
00:08:35,280 --> 00:08:38,510
So for example in this case, I might decide that I

165
00:08:38,510 --> 00:08:42,520
want to have only one test frame that tests my program

166
00:08:42,520 --> 00:08:44,420
with a file being empty and I can do the

167
00:08:44,420 --> 00:08:47,370
same for other choices. So basically I can continue adding this

168
00:08:47,370 --> 00:08:50,400
single constraint until I get down to the number of test

169
00:08:50,400 --> 00:08:53,410
frames and therefore the number of test cases that I want.

170
00:08:53,410 --> 00:08:57,770
So now let's go back once more to our console. And so now if we run

171
00:08:59,060 --> 00:09:04,450
using this file as input, you can see that we have 35 test frames generated. So

172
00:09:04,450 --> 00:09:07,750
this is a fairly low number of test cases, so we might decide that we want to

173
00:09:07,750 --> 00:09:13,380
go ahead and write these test frames to a file. So now let's open this file

174
00:09:15,990 --> 00:09:25,500
that we just generated. And as you can see here, I have exactly 35 test frames,

175
00:09:26,670 --> 00:09:30,900
as expected. Some of those correspond to the single and error cases. So in this

176
00:09:30,900 --> 00:09:33,330
case, the only choice that I have indicated

177
00:09:33,330 --> 00:09:35,690
is the one that corresponds to the single

178
00:09:35,690 --> 00:09:38,310
or error constraint. What is for the other

179
00:09:38,310 --> 00:09:42,170
ones? I actually have the whole test spec.

180
00:09:42,170 --> 00:09:45,530
So let's pick one just to give you an example.

181
00:09:45,530 --> 00:09:48,440
In this case, that's frame number 15 that will correspond to

182
00:09:48,440 --> 00:09:51,910
test case number 15. And here you can see that

183
00:09:51,910 --> 00:09:55,280
we have all the information. So this is a test specification.

184
00:09:55,280 --> 00:09:57,560
All the information that we need to generate the corresponding

185
00:09:57,560 --> 00:09:59,760
test. We know that we need a file that is not

186
00:09:59,760 --> 00:10:03,810
empty. That we need to have one occurrence of the pattern

187
00:10:03,810 --> 00:10:07,580
in the file. One occurrence of the pattern in one line.

188
00:10:08,680 --> 00:10:10,360
The position of the pattern in the file can

189
00:10:10,360 --> 00:10:13,740
be any position. The length of the pattern must

190
00:10:13,740 --> 00:10:16,640
be more than one character. The pattern should not

191
00:10:16,640 --> 00:10:20,140
be enclosed in quotes. There should be one white

192
00:10:20,140 --> 00:10:24,460
space, one quote within the pattern, and finally the

193
00:10:24,460 --> 00:10:27,230
file that would pass through the program should exist.

194
00:10:27,230 --> 00:10:29,680
So the file should be present. So I can

195
00:10:29,680 --> 00:10:33,950
easily transform all of this into an actual test case.

196
00:10:33,950 --> 00:10:35,540
And notice that even though we're not, we're not

197
00:10:35,540 --> 00:10:38,420
going to do it here. In cases like this, it might

198
00:10:38,420 --> 00:10:42,190
even be possible to automatically generate the test cases

199
00:10:42,190 --> 00:10:45,020
from the test specifications because, here for example, here it

200
00:10:45,020 --> 00:10:48,150
should be relatively straight forward to parse these test

201
00:10:48,150 --> 00:10:52,450
specifications and generate test cases accordingly. So, just to summarize,

202
00:10:52,450 --> 00:10:55,910
what we have done is to go from one high-level

203
00:10:55,910 --> 00:10:58,880
description of a program to a set of categories, partitions,

204
00:10:58,880 --> 00:11:01,820
and choices for that program. Then we have combined them

205
00:11:01,820 --> 00:11:04,930
in different ways, adding more and more constraints to reduce the

206
00:11:04,930 --> 00:11:07,600
number of combinations until we ended up with the right number

207
00:11:07,600 --> 00:11:09,650
of test cases, so the number of test cases that we

208
00:11:09,650 --> 00:11:14,630
were fine generating. We generated the corresponding test specifications. And at

209
00:11:14,630 --> 00:11:17,340
that point, we could just go ahead, generate the test case,

210
00:11:17,340 --> 00:11:20,660
and test our application. So, and you can see how this

211
00:11:20,660 --> 00:11:23,720
can result in a much more thorough testing of your application.

212
00:11:23,720 --> 00:11:27,890
Because instead of reading this description and just trying to come up with test

213
00:11:27,890 --> 00:11:33,900
cases for it, we can break down the process in steps that are easy to perform

214
00:11:33,900 --> 00:11:36,960
individually. They can be automated as much

215
00:11:36,960 --> 00:11:38,600
as possible. And they will end up with

216
00:11:38,600 --> 00:11:40,380
a set of test cases that will test

217
00:11:40,380 --> 00:11:42,790
all the interests and aspects of your application.