diff options
| author | Nguyễn Gia Phong <mcsinyx@disroot.org> | 2020-05-24 16:34:31 +0700 |
|---|---|---|
| committer | Nguyễn Gia Phong <mcsinyx@disroot.org> | 2020-05-24 16:34:31 +0700 |
| commit | b2d80610db6beda38573890ed169815e495bc663 (patch) | |
| tree | 176e1bca6fe644c619d53cf1c24682c244b79cf6 /usth/ICT2.7/P4L3 White-Box Testing Subtitles | |
| parent | 49376ab97c7427f1c1eca64072d1a934c2e52f50 (diff) | |
| download | cp-b2d80610db6beda38573890ed169815e495bc663.tar.gz | |
[usth/ICT2.7] Engineer software
Diffstat (limited to 'usth/ICT2.7/P4L3 White-Box Testing Subtitles')
30 files changed, 2946 insertions, 0 deletions
diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/1 - Lesson Overview - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/1 - Lesson Overview - lang_en_vs4.srt new file mode 100644 index 0000000..64f1576 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/1 - Lesson Overview - lang_en_vs4.srt @@ -0,0 +1,47 @@ +1 +00:00:00,110 --> 00:00:02,570 +In the previous lesson we discussed black box + +2 +00:00:02,570 --> 00:00:05,120 +testing, which is the testing performed based on a + +3 +00:00:05,120 --> 00:00:08,760 +description of the software, but without considering any of + +4 +00:00:08,760 --> 00:00:12,440 +software's internal details. In this lesson, we will discuss + +5 +00:00:12,440 --> 00:00:15,490 +the counterpart of black box testing, which is called + +6 +00:00:15,490 --> 00:00:20,590 +unsurprisingly white box testing or structural testing. And just + +7 +00:00:20,590 --> 00:00:22,780 +like we did for black box testing, we will + +8 +00:00:22,780 --> 00:00:25,740 +cover the main characteristics of white box testing and + +9 +00:00:25,740 --> 00:00:30,620 +this casts the most commonly used white box testing techniques. We + +10 +00:00:30,620 --> 00:00:33,650 +will conclude the lesson with the discussion of the main strengths and + +11 +00:00:33,650 --> 00:00:37,540 +limitations of white box testing. So that you will know, when to + +12 +00:00:37,540 --> 00:00:40,950 +use it? How to use it? And what to expect from it? diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/10 - Statement Coverage Quiz Solution - lang_en_vs7.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/10 - Statement Coverage Quiz Solution - lang_en_vs7.srt new file mode 100644 index 0000000..c18d579 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/10 - Statement Coverage Quiz Solution - lang_en_vs7.srt @@ -0,0 +1,31 @@ +1 +00:00:00,210 --> 00:00:01,900 +To get the answer to this question, I'm + +2 +00:00:01,900 --> 00:00:03,540 +going to ask you to be a little patient, and + +3 +00:00:03,540 --> 00:00:05,120 +to wait until the end of the lesson. + +4 +00:00:05,120 --> 00:00:06,792 +Because there are a couple of more topics that + +5 +00:00:06,792 --> 00:00:08,244 +I want to cover, before I actually get in + +6 +00:00:08,244 --> 00:00:10,210 +to this. Nevertheless, I wanted to ask you right + +7 +00:00:10,210 --> 00:00:11,880 +away, because I wanted you to think about + +8 +00:00:11,880 --> 00:00:13,560 +this, before you see the rest of the lesson. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/11 - Control Flow Graphs - lang_en_vs5.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/11 - Control Flow Graphs - lang_en_vs5.srt new file mode 100644 index 0000000..83d5860 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/11 - Control Flow Graphs - lang_en_vs5.srt @@ -0,0 +1,167 @@ +1 +00:00:00,320 --> 00:00:03,450 +Let's look at the code for PrintSum in a slightly different way + +2 +00:00:03,450 --> 00:00:06,140 +by making something explicit. If we go through the code, we can + +3 +00:00:06,140 --> 00:00:08,578 +see that the, the code does something in that case, if the + +4 +00:00:08,578 --> 00:00:11,720 +result greater then zero, does something else if the result is not + +5 +00:00:11,720 --> 00:00:14,890 +greater than zero but is less than zero, and otherwise in the + +6 +00:00:14,890 --> 00:00:17,800 +case in which neither of these two conditions is true. Nothing really + +7 +00:00:17,800 --> 00:00:19,090 +happens. So we're going to make that + +8 +00:00:19,090 --> 00:00:21,290 +explicit, we're going to say here, otherwise + +9 +00:00:21,290 --> 00:00:25,430 +do nothing, which is exactly our problem, the code does nothing, in this + +10 +00:00:25,430 --> 00:00:28,380 +case where it should do something. So now, let's look again in + +11 +00:00:28,380 --> 00:00:31,470 +our test cases, let's consider the first one, and I'm going to go a + +12 +00:00:31,470 --> 00:00:34,410 +little faster in this case, because we already saw what happens If + +13 +00:00:34,410 --> 00:00:37,890 +we execute the first test case, we get to this point, we execute + +14 +00:00:37,890 --> 00:00:40,710 +this statement, and then we just jump to the end, as we + +15 +00:00:40,710 --> 00:00:44,020 +saw. Now we, if we execute the second test case, we do the + +16 +00:00:44,020 --> 00:00:46,740 +same, we get to the else statement, the condition for the if is + +17 +00:00:46,740 --> 00:00:50,770 +true, and therefore we execute this statement. And we never reached this point + +18 +00:00:50,770 --> 00:00:53,320 +for either of the test cases. So how can we express + +19 +00:00:53,320 --> 00:00:56,470 +this? In order to do that, I'm going to introduce a very useful + +20 +00:00:56,470 --> 00:01:00,450 +concept. The concept of control flow graphs. The control flow graphs + +21 +00:01:00,450 --> 00:01:03,310 +is just a representation for the code that is very convenient when + +22 +00:01:03,310 --> 00:01:06,030 +we run our reason about the code and its structure. And + +23 +00:01:06,030 --> 00:01:09,910 +it's a fairly simple one that represents statement with notes and the + +24 +00:01:09,910 --> 00:01:12,900 +flow of control within the code with edges. So here's an + +25 +00:01:12,900 --> 00:01:16,180 +example of control flow graph for this code. There is the entry + +26 +00:01:16,180 --> 00:01:19,790 +point of the code right here, then our statement in which we + +27 +00:01:19,790 --> 00:01:23,320 +assign the result of A plus B to variable result. Our if + +28 +00:01:23,320 --> 00:01:25,720 +statement and as you can see the if statement it's got two + +29 +00:01:25,720 --> 00:01:28,930 +branches coming out of it, because based on the outcome of this + +30 +00:01:28,930 --> 00:01:32,376 +predicate we will go one way or the other. In fact normally + +31 +00:01:32,376 --> 00:01:36,000 +what we do, we will label this edges accordingly. So for example, + +32 +00:01:36,000 --> 00:01:38,580 +here we will say that this is the label to be executed + +33 +00:01:38,580 --> 00:01:41,470 +when the predicate is true. And this is the label that is executed + +34 +00:01:41,470 --> 00:01:44,120 +when the predicate is false. Now, at this point, similar + +35 +00:01:44,120 --> 00:01:47,080 +thing, statement five which corresponds with this one, we have another + +36 +00:01:47,080 --> 00:01:49,650 +if statement and if that statement is true, then we get + +37 +00:01:49,650 --> 00:01:51,830 +to this point and if it's false, we get to this + +38 +00:01:51,830 --> 00:01:54,370 +point. So as you can see, this graph represents my + +39 +00:01:54,370 --> 00:01:57,040 +code, in a much more intuitive way, because I can see + +40 +00:01:57,040 --> 00:02:00,650 +right away where the control flows, while I execute the code. + +41 +00:02:00,650 --> 00:02:01,730 +So we're going to use this + +42 +00:02:01,730 --> 00:02:04,430 +representation to introduce further coverage criteria. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/12 - Branch Coverage - lang_en_vs3.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/12 - Branch Coverage - lang_en_vs3.srt new file mode 100644 index 0000000..a1dedad --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/12 - Branch Coverage - lang_en_vs3.srt @@ -0,0 +1,267 @@ +1 +00:00:00,400 --> 00:00:03,780 +So now that we know what the CFG is, let's see how we can get into a count, + +2 +00:00:03,780 --> 00:00:07,860 +that else part, that is missing in our example code by introducing a new + +3 +00:00:07,860 --> 00:00:10,480 +kind of coverage. Branch coverage. As usual, + +4 +00:00:10,480 --> 00:00:13,550 +I'm going to describe branch coverage in terms of test requirements and + +5 +00:00:13,550 --> 00:00:17,100 +coverage measure. So starting from test requirements. The test requirement for + +6 +00:00:17,100 --> 00:00:21,080 +branch coverage are the branches in the program. In other words, + +7 +00:00:21,080 --> 00:00:25,240 +the goal of branch coverage is to execute all of the branches in the program. + +8 +00:00:25,240 --> 00:00:28,370 +The coverage measure is defined accordingly as the number of branches + +9 +00:00:28,370 --> 00:00:33,150 +executed by my test cases over the total number of branches in the program. And + +10 +00:00:33,150 --> 00:00:37,020 +let me remind you that branches are the outgoing edges from a decision point. + +11 +00:00:37,020 --> 00:00:40,410 +Therefore, an if statement, a switch statement, a while statement. + +12 +00:00:40,410 --> 00:00:44,400 +Any note in the c of g that has got more than one outgoing edge. + +13 +00:00:44,400 --> 00:00:48,170 +Those edges are called branches. So let's look at that using our example. So + +14 +00:00:48,170 --> 00:00:51,690 +now we're looking back at our printSum example. And in addition to the code, + +15 +00:00:51,690 --> 00:00:54,440 +I also want to represent the CFG for the program. So + +16 +00:00:54,440 --> 00:00:58,590 +let's start by looking at how many branches we have in our code. Which means how + +17 +00:00:58,590 --> 00:01:02,070 +many test requirements we have. And in this case there are two decision points. + +18 +00:01:02,070 --> 00:01:04,940 +The first one that corresponds to the first if, and the second one that + +19 +00:01:04,940 --> 00:01:10,100 +corresponds to the second if. So we have one, two, three, and four branches. So + +20 +00:01:10,100 --> 00:01:13,880 +now, let's bring back our current set of test cases. We had two test cases. + +21 +00:01:13,880 --> 00:01:17,440 +The one's that, with which we achieved a 100% statement coverage. And + +22 +00:01:17,440 --> 00:01:21,140 +let's see what happens in terms of branch coverage when we run these test cases. + +23 +00:01:21,140 --> 00:01:24,110 +I start from the first one, when we execute it, we follow the code, + +24 +00:01:24,110 --> 00:01:27,890 +we get to this decision point because the predicate in the if statement is true. + +25 +00:01:27,890 --> 00:01:30,770 +We follow the true branch, therefore we get here and then, + +26 +00:01:30,770 --> 00:01:34,970 +we exit from the program. So, in this case, we covered one of the branches, + +27 +00:01:34,970 --> 00:01:40,710 +which means that we got to 25% coverage. Now when we run the second test case, + +28 +00:01:40,710 --> 00:01:44,160 +again we follow this path. We get to this, the first if and + +29 +00:01:44,160 --> 00:01:47,730 +in this case the predicate of the if is false. Therefore, we go this way. + +30 +00:01:47,730 --> 00:01:51,160 +We reach the second predicate, the second if. The result is true, so + +31 +00:01:51,160 --> 00:01:55,490 +we follow the true branch and therefore, we cover these additional two branches. + +32 +00:01:55,490 --> 00:01:59,900 +So at this point, we are at 75% branch coverage. So what + +33 +00:01:59,900 --> 00:02:04,070 +happens is that we're missing this branch. For now, the inputs that we consider, + +34 +00:02:04,070 --> 00:02:07,740 +this branch is executed. Therefore, we need to add an additional test case. And + +35 +00:02:07,740 --> 00:02:11,600 +that this case that we need, is one for which this predicate is false and this + +36 +00:02:11,600 --> 00:02:15,680 +predicate is false. The simplest possibility in this case is the test case for + +37 +00:02:15,680 --> 00:02:20,170 +which A is equal to 0 and B is equal to 0. If we execute this test case, + +38 +00:02:20,170 --> 00:02:24,250 +our execution again followed this path, follows the fourth branch here. + +39 +00:02:24,250 --> 00:02:28,090 +And in this case, because result is not less than zero either, will follow + +40 +00:02:28,090 --> 00:02:33,090 +this branch as well. And therefore, we will reach our 100% branch coverage. And + +41 +00:02:33,090 --> 00:02:35,820 +this covered the problem. Something that I would like to clarify before we + +42 +00:02:35,820 --> 00:02:40,920 +move to the next topic, is that 100% coverage does not provide any guarantee of + +43 +00:02:40,920 --> 00:02:44,530 +finding the problems in the code. All we saw so far is the fact that by + +44 +00:02:44,530 --> 00:02:48,580 +testing more thoroughly we have more chances of finding a problem in the code. + +45 +00:02:48,580 --> 00:02:51,680 +But it doesn't matter which kind of coverage we utilize, and how much + +46 +00:02:51,680 --> 00:02:55,440 +coverage we achieve. There's always a chance that we might miss something. And + +47 +00:02:55,440 --> 00:02:59,760 +I will get back to this later on in the lesson. I just mentioned the fact that + +48 +00:02:59,760 --> 00:03:03,910 +we tested more fully when we went from statement coverage to branch coverage. + +49 +00:03:03,910 --> 00:03:06,900 +What does that mean exactly? To explain that, I'm going to introduce the concept + +50 +00:03:06,900 --> 00:03:11,420 +of test criteria subsumption. One test criteria subsumes another criteria when + +51 +00:03:11,420 --> 00:03:16,760 +all the tests widths that satisfy that criteria will also satisfy the other one. + +52 +00:03:16,760 --> 00:03:18,740 +So let me show you that with statement and + +53 +00:03:18,740 --> 00:03:23,220 +branch coverage. If we identify a test width that achieves 100% branch coverage, + +54 +00:03:23,220 --> 00:03:27,780 +the same test width will also achieve, necessarily, 100% statement coverage. + +55 +00:03:27,780 --> 00:03:31,290 +That's what happened for our example, and also what happens in general, + +56 +00:03:31,290 --> 00:03:35,660 +because branch coverage is a stronger criteria than statement coverage. + +57 +00:03:35,660 --> 00:03:39,670 +There is no way to cover all the branches without covering all the statements. + +58 +00:03:39,670 --> 00:03:43,480 +It is not true that any test results satisfies statement coverage will also + +59 +00:03:43,480 --> 00:03:47,040 +satisfy branch coverage. And, in fact, we just saw a counter example. + +60 +00:03:47,040 --> 00:03:50,569 +When we look at the printSum code. We had a test where there was achieving 100% + +61 +00:03:50,569 --> 00:03:53,910 +statement coverage and was not achieving 100% branch coverage. Therefore, + +62 +00:03:53,910 --> 00:03:58,301 +in this case we have a substantial relation in this direction. Branch coverage, + +63 +00:03:58,301 --> 00:04:02,860 +subsumes statement coverage. What it also means is that normally, or in general, + +64 +00:04:02,860 --> 00:04:06,910 +it is more expensive to achieve branch coverage than achieve statement coverage, + +65 +00:04:06,910 --> 00:04:09,490 +because achieving branch coverage requires the generation of + +66 +00:04:09,490 --> 00:04:13,770 +a larger number of test cases. So what this relation means is that + +67 +00:04:13,770 --> 00:04:17,779 +branch coverage is stronger than statement coverage but also more expensive. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/13 - Condition Coverage - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/13 - Condition Coverage - lang_en_vs4.srt new file mode 100644 index 0000000..bba72fc --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/13 - Condition Coverage - lang_en_vs4.srt @@ -0,0 +1,191 @@ +1 +00:00:00,470 --> 00:00:03,046 +What I'm going to do next is to introduce a few additional + +2 +00:00:03,046 --> 00:00:06,350 +coverage criteria using a slightly more complex example, but still a + +3 +00:00:06,350 --> 00:00:09,370 +pretty simple one. What I'm showing here is a program that + +4 +00:00:09,370 --> 00:00:12,145 +reads two real numbers, x and y. And then, if x + +5 +00:00:12,145 --> 00:00:15,148 +is equal to 0 or y is greater than 0, it + +6 +00:00:15,148 --> 00:00:19,360 +computes y as y divided by x. Otherwise, it computes x + +7 +00:00:19,360 --> 00:00:22,110 +as y plus 2, then it writes the value of x, + +8 +00:00:22,110 --> 00:00:25,730 +the value of y, and it terminates. Let's also introduce a CFG + +9 +00:00:25,730 --> 00:00:29,160 +for the program. As you can see, the CFG represents the statements + +10 +00:00:29,160 --> 00:00:31,900 +in the code and their control flow. And in this case, I made + +11 +00:00:31,900 --> 00:00:35,800 +explicit over the branches what are the conditions under which those branches + +12 +00:00:35,800 --> 00:00:38,530 +are taken to make it simpler to look at the example. Let's assume + +13 +00:00:38,530 --> 00:00:41,280 +that we have two tests for this code that are shown here. + +14 +00:00:41,280 --> 00:00:44,286 +For the first one, the inputs are 5 and 5. For the second + +15 +00:00:44,286 --> 00:00:47,610 +one, 5 and minus 5. If we consider branch coverage for this code + +16 +00:00:47,610 --> 00:00:50,990 +and we consider the two test cases, for the first one this condition + +17 +00:00:50,990 --> 00:00:53,440 +is true. Because x is not equal to 0 but y + +18 +00:00:53,440 --> 00:00:56,660 +is greater than 0. And therefore, we will follow this tree branch. + +19 +00:00:56,660 --> 00:00:59,640 +For the second one, the condition is false. Because x is not + +20 +00:00:59,640 --> 00:01:03,200 +equal to 0 and y is not greater than 0. Therefore, the + +21 +00:01:03,200 --> 00:01:05,990 +negation of it is true and we will follow this branch. In + +22 +00:01:05,990 --> 00:01:09,530 +other words, these two test cases achieve 100% branch coverage on this + +23 +00:01:09,530 --> 00:01:12,120 +code. If we look at the code though, we can see that + +24 +00:01:12,120 --> 00:01:15,879 +there is the possibility of making this code fail. Consider this statement, + +25 +00:01:15,879 --> 00:01:18,147 +if x is equal to 0, we could have a division + +26 +00:01:18,147 --> 00:01:21,710 +by 0. However, these two test cases, despite the fact that + +27 +00:01:21,710 --> 00:01:25,008 +they achieved 100% branch coverage, will not rebuild this problem. So + +28 +00:01:25,008 --> 00:01:27,564 +how can we be more thorough? I'll let you think about it + +29 +00:01:27,564 --> 00:01:29,964 +for a second, so think about how can we test more + +30 +00:01:29,964 --> 00:01:33,260 +thoroughly, in a more complete way, this code. So, in a way + +31 +00:01:33,260 --> 00:01:37,220 +that goes beyond branch coverage. And the answer is that we + +32 +00:01:37,220 --> 00:01:41,250 +can make each condition true and false. Instead of just considering the + +33 +00:01:41,250 --> 00:01:44,570 +whole predicate here. And that's exactly what is required by + +34 +00:01:44,570 --> 00:01:47,480 +the next criteria that we're going to consider which is condition + +35 +00:01:47,480 --> 00:01:49,880 +coverage. We're going to define it as usual in terms of + +36 +00:01:49,880 --> 00:01:53,070 +test requirements and coverage measure. In this case, the test + +37 +00:01:53,070 --> 00:01:57,540 +requirements for condition coverage are the individual conditions in the + +38 +00:01:57,540 --> 00:02:00,510 +program. So we want each condition in the program to + +39 +00:02:00,510 --> 00:02:03,900 +be both true and false first time execution. So the + +40 +00:02:03,900 --> 00:02:06,420 +way in which we can measure that is by measuring the + +41 +00:02:06,420 --> 00:02:09,150 +number of conditions that were both true and false + +42 +00:02:09,150 --> 00:02:11,580 +when we executed our tests over the total number + +43 +00:02:11,580 --> 00:02:14,180 +of conditions. And that gives us the percentage of + +44 +00:02:14,180 --> 00:02:17,270 +coverage that we achieved for condition coverage. Again, if you + +45 +00:02:17,270 --> 00:02:18,638 +want to look at this criteria in the form + +46 +00:02:18,638 --> 00:02:21,245 +of a question. The question would be, has each boolean + +47 +00:02:21,245 --> 00:02:25,480 +sub-expression, which means every condition in every predicate, evaluated + +48 +00:02:25,480 --> 00:02:28,010 +both to true and false when we run our tests. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/14 - Subsumption Quiz - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/14 - Subsumption Quiz - lang_en_vs4.srt new file mode 100644 index 0000000..de78093 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/14 - Subsumption Quiz - lang_en_vs4.srt @@ -0,0 +1,31 @@ +1 +00:00:00,080 --> 00:00:02,220 +So now that we introduced an additional criterion, + +2 +00:00:02,220 --> 00:00:04,290 +let's have a quiz to see whether everybody + +3 +00:00:04,290 --> 00:00:06,410 +understood the concept of subsumption. We know that + +4 +00:00:06,410 --> 00:00:09,490 +branch coverage subsumes statement coverage, but what about condition + +5 +00:00:09,490 --> 00:00:12,230 +coverage? Does it subsume branch coverage? Is it + +6 +00:00:12,230 --> 00:00:13,720 +the case that all of the test which + +7 +00:00:13,720 --> 00:00:16,980 +satisfy condition coverage will also satisfy branch coverage? + +8 +00:00:16,980 --> 00:00:19,160 +Think about it, and answer either yes or no. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/15 - Subsumption Quiz Solution - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/15 - Subsumption Quiz Solution - lang_en_vs4.srt new file mode 100644 index 0000000..87140c9 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/15 - Subsumption Quiz Solution - lang_en_vs4.srt @@ -0,0 +1,7 @@ +1 +00:00:00,170 --> 00:00:02,475 +In this case, the answer is no, and I'm + +2 +00:00:02,475 --> 00:00:04,650 +going to show you an example of this in a minute. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/16 - Branch and Condition Coverage - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/16 - Branch and Condition Coverage - lang_en_vs4.srt new file mode 100644 index 0000000..143a534 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/16 - Branch and Condition Coverage - lang_en_vs4.srt @@ -0,0 +1,143 @@ +1 +00:00:00,052 --> 00:00:03,407 +Let's go back to our test criteria subsumption representation, where + +2 +00:00:03,407 --> 00:00:06,770 +we already had branch coverage and statement coverage. In this case, + +3 +00:00:06,770 --> 00:00:09,350 +if we want to add condition coverage to this page, we + +4 +00:00:09,350 --> 00:00:12,050 +have to put it here on this side, with no relationship + +5 +00:00:12,050 --> 00:00:15,830 +of subsumption with either branch coverage or statement coverage, which + +6 +00:00:15,830 --> 00:00:19,270 +means that the criteria are not comparable. So now let's consider + +7 +00:00:19,270 --> 00:00:22,108 +again our last example, and let's use a different test + +8 +00:00:22,108 --> 00:00:25,060 +with this time. We still have two tests, but the first + +9 +00:00:25,060 --> 00:00:27,575 +one has x is equal to 0 and y is equal to minus + +10 +00:00:27,575 --> 00:00:30,370 +5. And the second one, x is equal to 5 and y is + +11 +00:00:30,370 --> 00:00:33,350 +equal to 5 as well. Let's see what happens in terms of condition + +12 +00:00:33,350 --> 00:00:36,740 +coverage, when we run these tests. If we look at the first condition, x + +13 +00:00:36,740 --> 00:00:39,820 +is equal to 0. It is both true, for the first test case, + +14 +00:00:39,820 --> 00:00:43,240 +and false for the second one. As for the second condition, y greater + +15 +00:00:43,240 --> 00:00:45,800 +than 0, it is false for the first test case and true for + +16 +00:00:45,800 --> 00:00:47,770 +the second one. Therefore we've achieved a + +17 +00:00:47,770 --> 00:00:50,400 +100% condition coverage with these two tests. + +18 +00:00:50,400 --> 00:00:54,020 +But what about branch coverage? If we consider the whole predicate, + +19 +00:00:54,020 --> 00:00:56,620 +instead of just the individual conditions, let's see what happens for the + +20 +00:00:56,620 --> 00:00:59,090 +two test cases. If we look at the first one, because + +21 +00:00:59,090 --> 00:01:02,200 +x is equal to 0, the overall predicate is true. As for + +22 +00:01:02,200 --> 00:01:05,750 +the second one, because the second condition is true, the overall + +23 +00:01:05,750 --> 00:01:09,110 +predicate is true. In other words, despite the fact that we're exercising + +24 +00:01:09,110 --> 00:01:12,520 +all the possible values for the two conditions, the overall predicate is + +25 +00:01:12,520 --> 00:01:16,540 +always true. And therefore, we're covering only one of the two branches. + +26 +00:01:16,540 --> 00:01:19,591 +So our coverage will be 50%. This is the reason + +27 +00:01:19,591 --> 00:01:22,231 +why normally the two criteria that we just saw, decision + +28 +00:01:22,231 --> 00:01:26,205 +coverage, and condition coverage are considered together. And the resulting + +29 +00:01:26,205 --> 00:01:30,320 +criterion is called branch and condition coverage, or also decision + +30 +00:01:30,320 --> 00:01:33,030 +and condition coverage. At this point the test requirements and + +31 +00:01:33,030 --> 00:01:35,590 +the coverage measure should be pretty straight forward, because they're + +32 +00:01:35,590 --> 00:01:38,120 +just considering the two criteria together. As far as the + +33 +00:01:38,120 --> 00:01:39,960 +requirements are concerned, they include + +34 +00:01:39,960 --> 00:01:41,880 +all the branches and individual conditions + +35 +00:01:41,880 --> 00:01:44,580 +in the program. Where as the coverage measure is computed + +36 +00:01:44,580 --> 00:01:49,180 +considering both coverage measures, both branch coverage and condition coverage. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/17 - Subsumption Quiz - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/17 - Subsumption Quiz - lang_en_vs4.srt new file mode 100644 index 0000000..9d99495 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/17 - Subsumption Quiz - lang_en_vs4.srt @@ -0,0 +1,15 @@ +1 +00:00:00,100 --> 00:00:02,810 +Let's have another quick quiz about Subsumption. If we + +2 +00:00:02,810 --> 00:00:06,720 +consider Branch and Condition Coverage, does it subsume Branch Coverage, + +3 +00:00:06,720 --> 00:00:09,512 +and therefore Statement Coverage? Or in other words, does branch + +4 +00:00:09,512 --> 00:00:12,750 +and condition coverage imply branch coverage? Answer yes, or no. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/18 - Subsumption Quiz Solution - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/18 - Subsumption Quiz Solution - lang_en_vs4.srt new file mode 100644 index 0000000..c836f04 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/18 - Subsumption Quiz Solution - lang_en_vs4.srt @@ -0,0 +1,19 @@ +1 +00:00:00,150 --> 00:00:01,880 +In this case it should be clear that the + +2 +00:00:01,880 --> 00:00:05,640 +answer is yes, because branch and condition coverage actually includes + +3 +00:00:05,640 --> 00:00:08,580 +branch coverage. Therefore, by definition, any test rate that + +4 +00:00:08,580 --> 00:00:10,505 +satisfies branch and condition coverage + +5 +00:00:10,505 --> 00:00:12,664 +will necessarily satisfy branch coverage. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/19 - Test Criteria Subsumption - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/19 - Test Criteria Subsumption - lang_en_vs4.srt new file mode 100644 index 0000000..f40ccdf --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/19 - Test Criteria Subsumption - lang_en_vs4.srt @@ -0,0 +1,47 @@ +1 +00:00:00,280 --> 00:00:03,400 +So we can now update our test criteria subsumption. We start + +2 +00:00:03,400 --> 00:00:06,700 +from this situation in which there are no relationship between condition + +3 +00:00:06,700 --> 00:00:09,650 +coverage, branch coverage and statement coverage. And when we add branch + +4 +00:00:09,650 --> 00:00:12,160 +and condition coverage, we can mark the fact that branch and + +5 +00:00:12,160 --> 00:00:14,470 +condition coverage subsumes branch coverage + +6 +00:00:14,470 --> 00:00:16,370 +and also subsumes condition coverage, for + +7 +00:00:16,370 --> 00:00:19,660 +the same reason. Therefore, this is a stronger criterion than both + +8 +00:00:19,660 --> 00:00:23,100 +branch coverage and condition coverage, and of course indirectly also soft + +9 +00:00:23,100 --> 00:00:25,330 +statement coverage. So once more, to make sure we are all + +10 +00:00:25,330 --> 00:00:29,255 +on the same page, if I develop a test rate that satisfies branch and condition + +11 +00:00:29,255 --> 00:00:31,380 +coverage, the same test will satisfy also + +12 +00:00:31,380 --> 00:00:35,210 +branch coverage, statement coverage and condition coverage, necessarily. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/2 - Overview - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/2 - Overview - lang_en_vs4.srt new file mode 100644 index 0000000..c032d40 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/2 - Overview - lang_en_vs4.srt @@ -0,0 +1,167 @@ +1 +00:00:00,420 --> 00:00:03,860 +In the last lesson, we talked about black-box testing or functional testing, + +2 +00:00:03,860 --> 00:00:06,110 +which is the kind of testing that you perform when you just + +3 +00:00:06,110 --> 00:00:09,860 +look at the description of the software. Today we're going to cover white-box + +4 +00:00:09,860 --> 00:00:12,490 +testing, which is the kind of testing that we perform when we + +5 +00:00:12,490 --> 00:00:15,800 +open up the box. We look inside the program, and we actually + +6 +00:00:15,800 --> 00:00:19,490 +test it based on its code. And there is one basic assumption + +7 +00:00:19,490 --> 00:00:22,320 +behind the idea of white-box testing, which is a very intuitive one, + +8 +00:00:22,320 --> 00:00:26,080 +and the assumption is that executing the faulty statement is a necessary + +9 +00:00:26,080 --> 00:00:29,380 +condition for revealing a fault. In other words, if there is + +10 +00:00:29,380 --> 00:00:31,690 +a bug in the program there is no way were going to be + +11 +00:00:31,690 --> 00:00:34,490 +able to find this bug or this fault, if we don't execute + +12 +00:00:34,490 --> 00:00:37,110 +the statement that contains it. Which makes a lot of sense. As + +13 +00:00:37,110 --> 00:00:40,240 +we did for black-box testing, we're going to start by summarizing what + +14 +00:00:40,240 --> 00:00:43,650 +are the main advantages of white-box testing. The main advantage is that + +15 +00:00:43,650 --> 00:00:48,050 +it's based on the code, and as such, the quality of white-box + +16 +00:00:48,050 --> 00:00:51,760 +testing can be measured objectively. And what I mean here by objectively + +17 +00:00:51,760 --> 00:00:54,390 +is that if you think about black-box testing In many + +18 +00:00:54,390 --> 00:00:57,630 +cases, there were subjective decisions, there were had to be made + +19 +00:00:57,630 --> 00:01:00,830 +in order to define tests in a black-box fashion. In the + +20 +00:01:00,830 --> 00:01:03,990 +case of white-box testing, because everything is based on the quota, + +21 +00:01:03,990 --> 00:01:07,965 +we don't have to make such subjective decisions. And similarly, because + +22 +00:01:07,965 --> 00:01:10,680 +white-box testing is based on the code, it can be measured + +23 +00:01:10,680 --> 00:01:14,250 +automatically. So we can build tools and actually there are tools, + +24 +00:01:14,250 --> 00:01:16,860 +and there's plenty of tools, that can be measured, the level + +25 +00:01:16,860 --> 00:01:19,670 +of white-box testing can be achieved in a fully automated way. + +26 +00:01:19,670 --> 00:01:21,620 +And we're going to see some of them in the course of the + +27 +00:01:21,620 --> 00:01:24,600 +class. Another advantage of white-box testing is that it can be + +28 +00:01:24,600 --> 00:01:27,700 +used to compare test suites. So if you have two alternative sets + +29 +00:01:27,700 --> 00:01:30,270 +of tests that you can use to assess the quality of + +30 +00:01:30,270 --> 00:01:34,360 +your software, white-box testing techniques can tell you which one of these + +31 +00:01:34,360 --> 00:01:37,580 +two test suites is likely to be more effective in testing + +32 +00:01:37,580 --> 00:01:42,350 +your code. And finally, white-box testing allows for covering the coded behavior + +33 +00:01:42,350 --> 00:01:44,470 +of the software. What that means is that if there is + +34 +00:01:44,470 --> 00:01:47,680 +some mistake in the code and is not obvious by looking at + +35 +00:01:47,680 --> 00:01:50,300 +the specification of the code, white box testing might be able + +36 +00:01:50,300 --> 00:01:53,330 +to catch it, because it tries to exercise the code. There's many + +37 +00:01:53,330 --> 00:01:56,430 +different kinds of white box testing, there are control flow based + +38 +00:01:56,430 --> 00:02:00,550 +techniques, data flow based techniques, and fault based techniques. And for each + +39 +00:02:00,550 --> 00:02:04,030 +one of these family of techniques there are many variations. So + +40 +00:02:04,030 --> 00:02:07,450 +this field is very, very broad. In this lesson we will talk + +41 +00:02:07,450 --> 00:02:09,110 +about white-box testing by focusing + +42 +00:02:09,110 --> 00:02:11,910 +mainly on control-flow based testing techniques. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/20 - B&C Coverage Quiz - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/20 - B&C Coverage Quiz - lang_en_vs4.srt new file mode 100644 index 0000000..6f5c9a1 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/20 - B&C Coverage Quiz - lang_en_vs4.srt @@ -0,0 +1,35 @@ +1 +00:00:00,130 --> 00:00:02,969 +So let's have another quiz using the example that we just + +2 +00:00:02,969 --> 00:00:06,660 +used. This one is about achieving 100% branch and condition coverage. So + +3 +00:00:06,660 --> 00:00:08,850 +let me bring back the two tests cases that we just saw, + +4 +00:00:08,850 --> 00:00:11,420 +and as we saw a few minutes ago, these tests cases do + +5 +00:00:11,420 --> 00:00:15,540 +not achieve 100% branch and condition coverage despite the fact that they + +6 +00:00:15,540 --> 00:00:19,430 +achieve 100% condition coverage. So both conditions are both true and false, + +7 +00:00:19,430 --> 00:00:21,820 +for these test cases. So what I want you to do is + +8 +00:00:21,820 --> 00:00:25,630 +to add a test case, to achieve 100% branch and condition coverages. + +9 +00:00:25,630 --> 00:00:29,740 +So specify the test case by writing here the value for x, and the value for y. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/21 - B&C Coverage Quiz Solution - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/21 - B&C Coverage Quiz Solution - lang_en_vs4.srt new file mode 100644 index 0000000..2b6fe4b --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/21 - B&C Coverage Quiz Solution - lang_en_vs4.srt @@ -0,0 +1,67 @@ +1 +00:00:00,190 --> 00:00:03,010 +Obviously there are many possible tests that we can use to + +2 +00:00:03,010 --> 00:00:05,910 +reach 100% branch and condition coverage. So I'm just going to show a + +3 +00:00:05,910 --> 00:00:08,740 +possible one, which is x equal to 3 and y is + +4 +00:00:08,740 --> 00:00:11,410 +equal to negative 2. If we specify this as case, you can + +5 +00:00:11,410 --> 00:00:14,900 +see that the overall condition is false, because neither x is + +6 +00:00:14,900 --> 00:00:17,990 +equal to 0 nor y is greater than 0. Therefore we will + +7 +00:00:17,990 --> 00:00:22,420 +follow the false, false branch, and achieve 100% branch and condition coverage + +8 +00:00:22,420 --> 00:00:25,400 +for this code. And we might require to be even more thorough, + +9 +00:00:25,400 --> 00:00:28,690 +that all the combinations of our conditions inside each + +10 +00:00:28,690 --> 00:00:31,790 +decision, inside each predicate, are tested. Which is what is + +11 +00:00:31,790 --> 00:00:36,010 +called, multiple condition coverage. But because of the way + +12 +00:00:36,010 --> 00:00:39,050 +this criterion is defined, it is combinatorial, becuse you have + +13 +00:00:39,050 --> 00:00:42,220 +to consider all the possible combinations of conditions. And + +14 +00:00:42,220 --> 00:00:45,210 +therefore it's extremely expensive, to the point of being impractical. + +15 +00:00:45,210 --> 00:00:47,610 +So instead of defining that criterion, we're going to find + +16 +00:00:47,610 --> 00:00:51,000 +another one which finds a good trade off between thoroughness + +17 +00:00:51,000 --> 00:00:52,815 +of the tests and their cost. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/22 - MC DC Coverage - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/22 - MC DC Coverage - lang_en_vs4.srt new file mode 100644 index 0000000..1a971f9 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/22 - MC DC Coverage - lang_en_vs4.srt @@ -0,0 +1,307 @@ +1 +00:00:00,120 --> 00:00:03,880 +And this criterion is called Modified Condition/Decision Coverage, + +2 +00:00:03,880 --> 00:00:07,750 +also called MC/DC. This criterion is very important because + +3 +00:00:07,750 --> 00:00:10,330 +it is often required for safety critical applications. + +4 +00:00:10,330 --> 00:00:13,920 +For example, the FAA, the Federal Aviation Administration, requires + +5 +00:00:13,920 --> 00:00:15,980 +for all the software that runs on commercial + +6 +00:00:15,980 --> 00:00:19,420 +airplanes to be tested according to the Modified Condition/Decision + +7 +00:00:19,420 --> 00:00:22,015 +Coverage. So what is the key idea behind the + +8 +00:00:22,015 --> 00:00:25,270 +MC/DC criterion? It is to test only the important + +9 +00:00:25,270 --> 00:00:28,710 +combinations of conditions instead of all of them, and limit + +10 +00:00:28,710 --> 00:00:32,189 +the testing cost by excluding the other combinations. And the way + +11 +00:00:32,189 --> 00:00:34,950 +in which it works is by extending branch and decision + +12 +00:00:34,950 --> 00:00:38,860 +coverage with the requirement that each condition should affect the decision + +13 +00:00:38,860 --> 00:00:41,940 +outcome independently. So let's see what this means with an + +14 +00:00:41,940 --> 00:00:45,030 +example that will also show you how you can reduce the + +15 +00:00:45,030 --> 00:00:46,940 +number of combinations in this way. I am going to + +16 +00:00:46,940 --> 00:00:50,460 +show you an example of how MC/DC works using this predicate + +17 +00:00:50,460 --> 00:00:53,870 +which consists of three conditions. a, b, and c, which are all + +18 +00:00:53,870 --> 00:00:57,070 +in and, so the overall predicate is a and b and c. + +19 +00:00:57,070 --> 00:00:58,980 +The first thing I'm going to do is to show you how + +20 +00:00:58,980 --> 00:01:03,370 +many test cases we will need to satisfy the multiple condition coverage + +21 +00:01:03,370 --> 00:01:06,440 +for this simple predicate. Which means, how many test cases we will + +22 +00:01:06,440 --> 00:01:09,780 +need to test all the possible combinations of true and false values + +23 +00:01:09,780 --> 00:01:12,740 +for these conditions. So I'm going to populate this table. And as you + +24 +00:01:12,740 --> 00:01:15,870 +can see, at the end we have eight test cases. Each test case + +25 +00:01:15,870 --> 00:01:18,650 +tests a different combination of values for a, b, and c. + +26 +00:01:18,650 --> 00:01:21,200 +I'm also showing, for each test case, the outcome of the overall + +27 +00:01:21,200 --> 00:01:23,930 +predicate. So, for example, if we look at the first one, the + +28 +00:01:23,930 --> 00:01:27,200 +first test case, will be such that a is true, b is + +29 +00:01:27,200 --> 00:01:30,210 +true, and c is true. And therefore, the overall outcome of + +30 +00:01:30,210 --> 00:01:34,360 +the predicate is true. Now lets consider the first condition, a. As + +31 +00:01:34,360 --> 00:01:36,300 +I said a minute ago, what we want to test are the + +32 +00:01:36,300 --> 00:01:39,160 +important combination. Which are the comibatinos + +33 +00:01:39,160 --> 00:01:41,930 +in which a single condition independently + +34 +00:01:41,930 --> 00:01:45,610 +effects the outcome of the overall predicate. So if we consider a + +35 +00:01:45,610 --> 00:01:48,490 +and we look at this possible set of this cases. Let's try to + +36 +00:01:48,490 --> 00:01:51,850 +find two test cases such that the only difference between the two + +37 +00:01:51,850 --> 00:01:54,960 +test cases is the value of a, and the overall outcome of the + +38 +00:01:54,960 --> 00:01:57,790 +predicate is different. If we look at the table, we can see + +39 +00:01:57,790 --> 00:02:00,920 +that this is true for test cases one and five. If we look + +40 +00:02:00,920 --> 00:02:04,030 +at these two cases, we can see that the overall of the predicate + +41 +00:02:04,030 --> 00:02:07,420 +in the two cases is true and false, and that the only difference + +42 +00:02:07,420 --> 00:02:10,720 +between the value of the conditions in the value of a. So + +43 +00:02:10,720 --> 00:02:13,990 +these test cases satisfy exactly what we wanted. There are two test + +44 +00:02:13,990 --> 00:02:18,270 +cases in which the value of a independently decides the overall value + +45 +00:02:18,270 --> 00:02:21,380 +of the predicate. What we do, therefore, is to add these first + +46 +00:02:21,380 --> 00:02:24,740 +two test cases to our set of tests down here. Now let's + +47 +00:02:24,740 --> 00:02:27,520 +focus on b and let's try to find two test cases such + +48 +00:02:27,520 --> 00:02:30,280 +that the value of b is the only value that changes between + +49 +00:02:30,280 --> 00:02:32,450 +the two test cases, but the overall value of the predicate is + +50 +00:02:32,450 --> 00:02:34,980 +different, the same thing we did for a. And in this case, + +51 +00:02:34,980 --> 00:02:37,610 +we can see that if we select test case number one, and test + +52 +00:02:37,610 --> 00:02:40,420 +case number three, we have exactly that situation. b is true in the + +53 +00:02:40,420 --> 00:02:44,250 +first case, false in the second one, a and c don't change, but + +54 +00:02:44,250 --> 00:02:46,830 +the overall value of the predicate changes. And now you can notice + +55 +00:02:46,830 --> 00:02:50,720 +something else. That even though we selected two test cases, tested two values, + +56 +00:02:50,720 --> 00:02:53,950 +one we already had. So, we only need three test cases overall to + +57 +00:02:53,950 --> 00:02:57,730 +test a and b according to MC/DC. Now, let's look at our last + +58 +00:02:57,730 --> 00:03:00,510 +condition, c. At this point, we know the game, so we + +59 +00:03:00,510 --> 00:03:02,780 +just have to look for two test cases that satisfy our + +60 +00:03:02,780 --> 00:03:06,380 +requirements. And in this case, one and two are suitable candidates. + +61 +00:03:06,380 --> 00:03:08,820 +And once more, because we already have one, we just have to + +62 +00:03:08,820 --> 00:03:11,070 +add two to our list. So as you can see from + +63 +00:03:11,070 --> 00:03:14,730 +this example, we went from having eight test cases needed to cover + +64 +00:03:14,730 --> 00:03:18,600 +all the possible combinations of conditions to only four test cases + +65 +00:03:18,600 --> 00:03:22,880 +to satisfy the MC/DC criteria. So let's see where MC/DC stands in + +66 +00:03:22,880 --> 00:03:25,510 +our substantion hierarchy. This is what we had so far + +67 +00:03:25,510 --> 00:03:28,430 +in the hierarchy and this is where the MC/DC criterion will + +68 +00:03:28,430 --> 00:03:31,930 +stand. MC/DC criterion is stronger than branch and condition coverage. + +69 +00:03:31,930 --> 00:03:35,210 +Why? Because it requires every single condition to be true and + +70 +00:03:35,210 --> 00:03:38,400 +false. And therefore, this section was a condition coverage criteria. + +71 +00:03:38,400 --> 00:03:41,610 +And it also requires every predicate to be true and false + +72 +00:03:41,610 --> 00:03:44,425 +and therefore, this section is branch coverage. And in addition, + +73 +00:03:44,425 --> 00:03:47,790 +it's got the additional requirements that the true and false values, + +74 +00:03:47,790 --> 00:03:50,330 +all the conditions have to also decide the overall + +75 +00:03:50,330 --> 00:03:53,190 +value of the predicate. So it's stronger. Which is more + +76 +00:03:53,190 --> 00:03:56,130 +thorough than branch and condition coverage and, as usual, + +77 +00:03:56,130 --> 00:03:59,380 +also stronger than branch coverage, statement coverage, and condition coverage. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/23 - Other Criteria - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/23 - Other Criteria - lang_en_vs4.srt new file mode 100644 index 0000000..27bf0fe --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/23 - Other Criteria - lang_en_vs4.srt @@ -0,0 +1,275 @@ +1 +00:00:00,120 --> 00:00:02,900 +As I mentioned at the beginning of the class, there are many, many, + +2 +00:00:02,900 --> 00:00:06,150 +many, white box criteria. And we're not going to have time to cover them + +3 +00:00:06,150 --> 00:00:08,000 +all. So what I like to do now is just to give you + +4 +00:00:08,000 --> 00:00:11,290 +the flavor, of some other criteria by just mentioning them, and saying a + +5 +00:00:11,290 --> 00:00:13,740 +couple of things, about the way they work. And the first one I + +6 +00:00:13,740 --> 00:00:15,370 +want to mention is path coverage. And + +7 +00:00:15,370 --> 00:00:17,800 +in path coverage, the test requirements are + +8 +00:00:17,800 --> 00:00:21,220 +all the paths in the program. So what that means is that to + +9 +00:00:21,220 --> 00:00:25,250 +satisfy this criteria, we need to generate enough test cases such that all + +10 +00:00:25,250 --> 00:00:27,230 +the paths in my program are covered. As you can + +11 +00:00:27,230 --> 00:00:31,970 +imagine this is incredibly expensive because any nontrivial program has got + +12 +00:00:31,970 --> 00:00:34,280 +a virtual infinite number of paths and therefore we will + +13 +00:00:34,280 --> 00:00:36,910 +need a virtual infinite number of test cases to satisfy the + +14 +00:00:36,910 --> 00:00:39,950 +path coverage criteria. Another family of criteria I want to mention + +15 +00:00:39,950 --> 00:00:43,970 +are data-flow coverage criteria and in data-flow coverage criteria, the focus + +16 +00:00:43,970 --> 00:00:47,480 +shifts from the coverage of individual elements in the code to + +17 +00:00:47,480 --> 00:00:50,320 +the coverage of pairs of elements. And in particular the coverage + +18 +00:00:50,320 --> 00:00:53,770 +of Statements, in which the content of some memory locations + +19 +00:00:53,770 --> 00:00:56,920 +modified, and statements in which the content of the same + +20 +00:00:56,920 --> 00:00:59,860 +memory location is used. So in this way, our test + +21 +00:00:59,860 --> 00:01:03,030 +will exercise the assignments of values to memory, and the + +22 +00:01:03,030 --> 00:01:06,020 +usage of those assignments,. Finally, I want to mention mutation + +23 +00:01:06,020 --> 00:01:09,150 +coverage. And this is a fairly different and new idea, + +24 +00:01:09,150 --> 00:01:11,910 +so the key concept in mutation coverage is that we + +25 +00:01:11,910 --> 00:01:16,156 +want to evaluate the goodness of our test by modifying the code. + +26 +00:01:16,156 --> 00:01:18,704 +For example here in this small I'm, I'm showing that I might + +27 +00:01:18,704 --> 00:01:21,820 +change it. An if statement from K greater than 9, to K + +28 +00:01:21,820 --> 00:01:24,310 +greater than or equal to 9. And the reason why I want to + +29 +00:01:24,310 --> 00:01:27,750 +do that, is that if I generate enough mutants, enough variation of + +30 +00:01:27,750 --> 00:01:30,340 +my program, then I can use them to assess how good are + +31 +00:01:30,340 --> 00:01:34,330 +my tests at distinguishing the original program and the mutants. And because + +32 +00:01:34,330 --> 00:01:37,430 +I'm changing the code based on the way I expect to introduce + +33 +00:01:37,430 --> 00:01:41,370 +errors in the code, the more my test can identify mutants, the better + +34 +00:01:41,370 --> 00:01:44,140 +they are at identifying real faults. This is a very high + +35 +00:01:44,140 --> 00:01:46,470 +level [UNKNOWN], but just to give you the intuition and the idea + +36 +00:01:46,470 --> 00:01:50,210 +behind these criteria, and I'm going to provide as usual, additional pointers + +37 +00:01:50,210 --> 00:01:52,930 +to go in more depth on these topics in the notes of + +38 +00:01:52,930 --> 00:01:55,170 +the class. So now, let's go back to our test criteria + +39 +00:01:55,170 --> 00:01:59,310 +subsumption hierarchy, and see how all of these criteria fit together in + +40 +00:01:59,310 --> 00:02:02,663 +this context. Let me start with multiple condition coverage. As we saw, + +41 +00:02:02,663 --> 00:02:06,740 +MC/DC is sort of as [UNKNOWN] way of doing multiple condition coverage + +42 +00:02:06,740 --> 00:02:09,229 +in the sense that it doesn't consider all the combinations, + +43 +00:02:09,229 --> 00:02:11,460 +but only the ones that are more likely to matter. + +44 +00:02:11,460 --> 00:02:14,930 +And as such, MC/DC exercises a subset of the elements + +45 +00:02:14,930 --> 00:02:16,220 +of the multiple condition coverage + +46 +00:02:16,220 --> 00:02:18,760 +exercises. And therefore, multiple condition coverage + +47 +00:02:18,760 --> 00:02:22,060 +is more thorough than MC/DC and subsumption, and it's also + +48 +00:02:22,060 --> 00:02:26,300 +as we saw incredibly more expensive. Path coverage subsumes branch coverage, + +49 +00:02:26,300 --> 00:02:28,090 +because if I cover all of the paths in the + +50 +00:02:28,090 --> 00:02:32,140 +code, I necessarily cover all the branches. However, it doesn't subsume + +51 +00:02:32,140 --> 00:02:34,130 +multiple condition coverage, MC/CD, or + +52 +00:02:34,130 --> 00:02:35,910 +branch and condition coverage. Because this + +53 +00:02:35,910 --> 00:02:38,590 +criteria, have additional requirements involving + +54 +00:02:38,590 --> 00:02:39,940 +the conditions of the predicate that + +55 +00:02:39,940 --> 00:02:42,730 +path coverage does not have. As for data-flow coverage criteria and + +56 +00:02:42,730 --> 00:02:45,010 +mutation coverage criteria, there really no + +57 +00:02:45,010 --> 00:02:46,860 +relation with the other criteria, because + +58 +00:02:46,860 --> 00:02:49,710 +they look at different aspects, of the code. So they're not + +59 +00:02:49,710 --> 00:02:52,220 +really comparable, and therefore we're just going to put them on the + +60 +00:02:52,220 --> 00:02:55,000 +side, without any relationship with the other ones. The reason why + +61 +00:02:55,000 --> 00:02:57,300 +I want to represent them all anyways is because I want to make + +62 +00:02:57,300 --> 00:02:59,950 +an important distinction between these different criteria. And + +63 +00:02:59,950 --> 00:03:02,690 +that's the distinction between practical criteria, which are + +64 +00:03:02,690 --> 00:03:05,490 +criteria that are actually not too expansive to + +65 +00:03:05,490 --> 00:03:08,790 +be used in real scenarios. And theoretical criteria, which + +66 +00:03:08,790 --> 00:03:11,590 +are criteria that are useful in theory from + +67 +00:03:11,590 --> 00:03:14,140 +the conceptual standpoint, but they're not really applicable + +68 +00:03:14,140 --> 00:03:16,540 +in practice because they're too expansive, because they + +69 +00:03:16,540 --> 00:03:19,460 +require basically too many test cases to be satisfied. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/24 - Review Quiz - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/24 - Review Quiz - lang_en_vs4.srt new file mode 100644 index 0000000..3f1d6ee --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/24 - Review Quiz - lang_en_vs4.srt @@ -0,0 +1,63 @@ +1 +00:00:00,160 --> 00:00:04,260 +White box testing, in general, and coverage criteria in particular, involve some + +2 +00:00:04,260 --> 00:00:07,550 +subtle concepts, so before I conclude this lesson, I want to have a + +3 +00:00:07,550 --> 00:00:10,960 +few more quizzes to make sure that we all understand these concepts. + +4 +00:00:10,960 --> 00:00:13,610 +The first one involves a very simple piece of code, a straight line + +5 +00:00:13,610 --> 00:00:16,540 +of code, three statements, in which we simply read an integer and + +6 +00:00:16,540 --> 00:00:20,180 +then prints 10 divided by the value of that integer minus 3. Now, + +7 +00:00:20,180 --> 00:00:22,250 +let's imagine that we have a test where there consists of three + +8 +00:00:22,250 --> 00:00:25,250 +test cases for this code, and what I'm showing in the test cases + +9 +00:00:25,250 --> 00:00:28,580 +is the input and the expected output. So for the first one, + +10 +00:00:28,580 --> 00:00:31,280 +the input is 1, and I expect the output to be minus 5. + +11 +00:00:31,280 --> 00:00:34,480 +For the second one, the input is minus 1, I'm expecting to + +12 +00:00:34,480 --> 00:00:37,120 +have 2.5. And for the third one, the input is 0, and I'm + +13 +00:00:37,120 --> 00:00:42,440 +expecting to have minus 3.3333 as the result. Now the first question + +14 +00:00:42,440 --> 00:00:45,180 +I want to ask, is if we considered this test suite, and we + +15 +00:00:45,180 --> 00:00:48,050 +run it on the code, does it achieve path coverage? And remember + +16 +00:00:48,050 --> 00:00:50,960 +that path coverage is one of the strongest coverage criteria that we saw. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/25 - Review Quiz Solution - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/25 - Review Quiz Solution - lang_en_vs4.srt new file mode 100644 index 0000000..19d1177 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/25 - Review Quiz Solution - lang_en_vs4.srt @@ -0,0 +1,7 @@ +1 +00:00:00,180 --> 00:00:04,040 +And the answer in this case is clearly yes. In fact, any input will really + +2 +00:00:04,040 --> 00:00:07,430 +achieve path coverage for this code because there is only one path in the code. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/26 - Review Quiz - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/26 - Review Quiz - lang_en_vs4.srt new file mode 100644 index 0000000..a07e712 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/26 - Review Quiz - lang_en_vs4.srt @@ -0,0 +1,15 @@ +1 +00:00:00,108 --> 00:00:03,890 +But now want to ask a different question which is, does this test reveal + +2 +00:00:03,890 --> 00:00:07,760 +the fault at line three? Clearly, here there is a possible problem, because + +3 +00:00:07,760 --> 00:00:10,440 +if we pass the right input, we can have a division by zero. + +4 +00:00:10,440 --> 00:00:13,760 +Is that revealed if we run this test within the code? Yes or no? diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/27 - Review Quiz Solution - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/27 - Review Quiz Solution - lang_en_vs4.srt new file mode 100644 index 0000000..9945694 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/27 - Review Quiz Solution - lang_en_vs4.srt @@ -0,0 +1,55 @@ +1 +00:00:00,340 --> 00:00:03,581 +And the answer is no. So even path coverage's stronger criteria + +2 +00:00:03,581 --> 00:00:06,376 +that we saw is not enough to reveal this problem. And why + +3 +00:00:06,376 --> 00:00:08,526 +is that? So let me go back to a concept that + +4 +00:00:08,526 --> 00:00:12,491 +I mentioned in a previous lesson, the cost of exhaustive testing. Exhaustive + +5 +00:00:12,491 --> 00:00:15,236 +testing is really the only way in which we can exercise + +6 +00:00:15,236 --> 00:00:18,230 +all of the possible behaviors of a program. So we can say + +7 +00:00:18,230 --> 00:00:21,884 +even though it might sound like topology, that only exhaustive testing is + +8 +00:00:21,884 --> 00:00:25,710 +exhausted. All the coverage criteria that we saw are just process and + +9 +00:00:25,710 --> 00:00:30,900 +just approximation of a complete test. So test is a best effort kind + +10 +00:00:30,900 --> 00:00:33,180 +of activity. Coverage criteria help you assess + +11 +00:00:33,180 --> 00:00:35,430 +how well you tested but once more, + +12 +00:00:35,430 --> 00:00:40,060 +test can only reveal issues, can only reveal problems. You can never show the + +13 +00:00:40,060 --> 00:00:42,530 +absence of problems. And that's something is + +14 +00:00:42,530 --> 00:00:44,377 +important to remember when we do testing. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/28 - Review Quiz - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/28 - Review Quiz - lang_en_vs4.srt new file mode 100644 index 0000000..d6dd40a --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/28 - Review Quiz - lang_en_vs4.srt @@ -0,0 +1,31 @@ +1 +00:00:00,290 --> 00:00:03,220 +Now let's do another quiz considering a slightly different piece + +2 +00:00:03,220 --> 00:00:05,050 +of code. In this case we have five layers of + +3 +00:00:05,050 --> 00:00:08,684 +code. We have a variable i, a variable j. Variable + +4 +00:00:08,684 --> 00:00:11,620 +j is read from the input. And then based on + +5 +00:00:11,620 --> 00:00:14,500 +this predicate, we either print the value of i or + +6 +00:00:14,500 --> 00:00:16,850 +simply exit. And what I want to know from you is + +7 +00:00:16,850 --> 00:00:19,990 +whether you can create a test suite to achieve statement + +8 +00:00:19,990 --> 00:00:22,660 +coverage for this simple piece of code. Yes or no? diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/29 - Review Quiz Solution - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/29 - Review Quiz Solution - lang_en_vs4.srt new file mode 100644 index 0000000..1e3228c --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/29 - Review Quiz Solution - lang_en_vs4.srt @@ -0,0 +1,119 @@ +1 +00:00:00,230 --> 00:00:02,520 +And the answer in this case is no. And + +2 +00:00:02,520 --> 00:00:05,480 +the reason for this is because this code is + +3 +00:00:05,480 --> 00:00:08,950 +unreachable. This is dead code because no matter the + +4 +00:00:08,950 --> 00:00:12,560 +value of j, this condition will always be false because + +5 +00:00:12,560 --> 00:00:14,770 +i will always be 0. And notice that this + +6 +00:00:14,770 --> 00:00:17,670 +is a small example, but another important truth is + +7 +00:00:17,670 --> 00:00:22,520 +that any non-trivial program contains dead or unreachable code, + +8 +00:00:22,520 --> 00:00:25,610 +code that no matter how well we test our system, + +9 +00:00:25,610 --> 00:00:28,450 +we will never be able to exercise. Why is that? Various + +10 +00:00:28,450 --> 00:00:31,030 +reasons. For example, defensive programming or, + +11 +00:00:31,030 --> 00:00:33,600 +for example, developing for future releases. + +12 +00:00:33,600 --> 00:00:36,320 +So there might be pieces of code that are added, but they're + +13 +00:00:36,320 --> 00:00:39,380 +still not activated. They're still not invoked by any other part of + +14 +00:00:39,380 --> 00:00:41,970 +the code because of modifications to the code. There are some + +15 +00:00:41,970 --> 00:00:44,480 +parts that were executed before and in the new version of the + +16 +00:00:44,480 --> 00:00:47,915 +program, they are no longer exercised. They are no longer executable. But + +17 +00:00:47,915 --> 00:00:50,760 +they still remain in the code base. And this is a very, + +18 +00:00:50,760 --> 00:00:54,560 +very normal situation for this reason and many more. And this is + +19 +00:00:54,560 --> 00:00:56,790 +an important concept because this affects + +20 +00:00:56,790 --> 00:00:58,760 +the very meaning of coverage measures. If + +21 +00:00:58,760 --> 00:01:01,225 +there is some unreachable code, we will never be able to reach + +22 +00:01:01,225 --> 00:01:04,709 +100% code coverage. And in fact, if you remember, at the beginning of + +23 +00:01:04,709 --> 00:01:07,380 +the lesson, we discussed this concept and asked you why you think + +24 +00:01:07,380 --> 00:01:10,470 +that, in the industry, the target for coverage is not 100% but less + +25 +00:01:10,470 --> 00:01:12,850 +that that. And that's the answer, to account for the fact that there + +26 +00:01:12,850 --> 00:01:15,890 +are infeasibility problems that I have to take into account when I test + +27 +00:01:15,890 --> 00:01:19,920 +the code, infeasible paths, unexecutable statements, conditions that can never + +28 +00:01:19,920 --> 00:01:22,780 +be true, and so on. So most criteria, if not + +29 +00:01:22,780 --> 00:01:24,620 +all, are affected, and we need to take that into + +30 +00:01:24,620 --> 00:01:27,270 +account when we try to achieve a given coverage target. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/3 - Coverage Criteria Intro - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/3 - Coverage Criteria Intro - lang_en_vs4.srt new file mode 100644 index 0000000..b1a38a7 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/3 - Coverage Criteria Intro - lang_en_vs4.srt @@ -0,0 +1,171 @@ +1 +00:00:00,320 --> 00:00:04,059 +So let's start our lesson on white docs testing by considering again the + +2 +00:00:04,059 --> 00:00:07,470 +program PrintSum. If you remember, this is the same program that we used when + +3 +00:00:07,470 --> 00:00:10,300 +we were talking about black box testing. It's the program that takes two + +4 +00:00:10,300 --> 00:00:14,050 +integers, A and B, and produces, as a result, the sum of the two. + +5 +00:00:14,050 --> 00:00:16,570 +And when we were looking at this problem in the context of black + +6 +00:00:16,570 --> 00:00:17,760 +box testing, we did not look at + +7 +00:00:17,760 --> 00:00:19,700 +implementation. But that's exactly what we're going to + +8 +00:00:19,700 --> 00:00:22,090 +do now. So we're going to open the box and look at how the + +9 +00:00:22,090 --> 00:00:25,700 +code is implemented. And as you can see, the programmer was kind of creative. + +10 +00:00:25,700 --> 00:00:28,670 +Because instead of just adding the two numbers and printing them, he + +11 +00:00:28,670 --> 00:00:32,080 +or she also decided to print them in a specific color depending + +12 +00:00:32,080 --> 00:00:36,840 +on whether they were positive numbers or negative numbers. So positive results + +13 +00:00:36,840 --> 00:00:40,040 +are printed in red and negative results are printed in blue. And as + +14 +00:00:40,040 --> 00:00:41,970 +you can see by looking at the code we can see some + +15 +00:00:41,970 --> 00:00:45,150 +interesting cases that we might want to test. For instance you can + +16 +00:00:45,150 --> 00:00:48,190 +see that there are two decisions made here. So we might decide + +17 +00:00:48,190 --> 00:00:51,100 +that this is an interesting case and therefore we want to test it. + +18 +00:00:51,100 --> 00:00:53,770 +Similarly we might look at this other case and we might + +19 +00:00:53,770 --> 00:00:56,465 +also decide that this is another interesting case and therefore we + +20 +00:00:56,465 --> 00:00:58,970 +want to test this one as well. So let's discuss this + +21 +00:00:58,970 --> 00:01:01,480 +in a slightly more formal way by introducing the concept of + +22 +00:01:01,480 --> 00:01:05,110 +coverage criteria which are really the essence of why box testing. + +23 +00:01:05,110 --> 00:01:08,930 +First of all coverage criteria are defined in terms of test + +24 +00:01:08,930 --> 00:01:13,460 +requirements where test requirements are the elements, the entities in the + +25 +00:01:13,460 --> 00:01:16,250 +code that we need to exercise. That we need to execute + +26 +00:01:16,250 --> 00:01:18,690 +in order to satisfy the criteria. And we'll see plenty + +27 +00:01:18,690 --> 00:01:21,430 +of examples of that. And normally, when I apply a coverage + +28 +00:01:21,430 --> 00:01:24,810 +criterion, my result is a set of test specifications. And we + +29 +00:01:24,810 --> 00:01:26,540 +already saw test specifications. Those + +30 +00:01:26,540 --> 00:01:29,540 +are basically descriptions, specifications, of how + +31 +00:01:29,540 --> 00:01:32,786 +the tests should be in order to satisfy the requirements. And + +32 +00:01:32,786 --> 00:01:36,210 +they also result in actual test cases, which are instantiations of + +33 +00:01:36,210 --> 00:01:39,470 +the test specifications. And again this is exactly analogous to what + +34 +00:01:39,470 --> 00:01:41,830 +we saw when we were talking about the black box testing. + +35 +00:01:41,830 --> 00:01:43,960 +So let's see what this means by going back to our + +36 +00:01:43,960 --> 00:01:46,680 +example. A minute ago, we looked at the print sum code + +37 +00:01:46,680 --> 00:01:50,000 +and we identified two interesting cases for our code. And those + +38 +00:01:50,000 --> 00:01:53,430 +are exactly our test requirements. So we have a first test + +39 +00:01:53,430 --> 00:01:57,840 +requirement here, which is the execution of this particular statement and + +40 +00:01:57,840 --> 00:02:01,500 +a second requirement here and this one corresponds to the execution + +41 +00:02:01,500 --> 00:02:04,580 +of this other statement. So for this example there are two + +42 +00:02:04,580 --> 00:02:06,920 +things that we need to do in order to satisfy our + +43 +00:02:06,920 --> 00:02:10,100 +coverage requirements. Execute this statement and execute this statement. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/30 - Summary - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/30 - Summary - lang_en_vs4.srt new file mode 100644 index 0000000..264491d --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/30 - Summary - lang_en_vs4.srt @@ -0,0 +1,127 @@ +1 +00:00:00,190 --> 00:00:02,800 +Now let me conclude the lesson by summarizing a few important + +2 +00:00:02,800 --> 00:00:06,840 +aspects of white-box testing. The first important aspect is that white-box + +3 +00:00:06,840 --> 00:00:09,770 +testing works on a formal model. The code itself are models + +4 +00:00:09,770 --> 00:00:12,430 +derived from the code. So when we do white-box testing, we + +5 +00:00:12,430 --> 00:00:15,160 +don't need to make subjective decision, for example, on the level + +6 +00:00:15,160 --> 00:00:18,890 +of obstruction of our models. Normally, we simply represent what's there. + +7 +00:00:18,890 --> 00:00:22,010 +And so what we will obtain are objective, results and objective + +8 +00:00:22,010 --> 00:00:25,400 +measures. As I also said at the beginning, coverage criteria allows + +9 +00:00:25,400 --> 00:00:28,790 +us to compare different test suites, different sets of tests, + +10 +00:00:28,790 --> 00:00:31,780 +because I can measure the coverage achieved by one test suite + +11 +00:00:31,780 --> 00:00:34,280 +and by the other, and then decide which one to use + +12 +00:00:34,280 --> 00:00:37,400 +based on this measure. And again, remember, these measures aren't perfect, + +13 +00:00:37,400 --> 00:00:40,700 +but they at least give you an objective number, an objective + +14 +00:00:40,700 --> 00:00:44,470 +measure of the likely effectiveness of your tests. So even though + +15 +00:00:44,470 --> 00:00:48,130 +achieving 100% coverage does not mean that you identify all the + +16 +00:00:48,130 --> 00:00:50,710 +problems in the code for sure. If your level of coverage + +17 +00:00:50,710 --> 00:00:53,700 +is 10%, for example, for stemen coverage. That means that + +18 +00:00:53,700 --> 00:00:57,360 +you haven't exercised 90% of your code, and therefore the trouble + +19 +00:00:57,360 --> 00:01:00,565 +is in a piece of software that is inadequately tested + +20 +00:01:00,565 --> 00:01:03,600 +and likely to be of inadequate quality. We also saw that + +21 +00:01:03,600 --> 00:01:07,370 +there are two broad classes of coverage criteria, practical criteria + +22 +00:01:07,370 --> 00:01:10,430 +that we can actually use, and theoretical criteria that are interesting + +23 +00:01:10,430 --> 00:01:13,410 +from a conceptual standpoint, but that they are totally impractical. + +24 +00:01:13,410 --> 00:01:16,170 +They are too expensive to be used on real world software. + +25 +00:01:16,170 --> 00:01:18,350 +And finally, as we also said at the beginning, + +26 +00:01:18,350 --> 00:01:20,820 +one of the great things about white box testing and + +27 +00:01:20,820 --> 00:01:23,910 +coverage criteria, is that they are fully automatable. There are + +28 +00:01:23,910 --> 00:01:27,380 +tools that can take your code, instrument it automatically. And + +29 +00:01:27,380 --> 00:01:29,050 +when you run your test cases, they will tell + +30 +00:01:29,050 --> 00:01:31,330 +you, what is the level of coverage that you achieve + +31 +00:01:31,330 --> 00:01:34,410 +with your test at no cost for you. So there's + +32 +00:01:34,410 --> 00:01:36,763 +really no reason not to measure coverage of your code. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/4 - Coverage Criteria Intro Quiz - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/4 - Coverage Criteria Intro Quiz - lang_en_vs4.srt new file mode 100644 index 0000000..2adce93 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/4 - Coverage Criteria Intro Quiz - lang_en_vs4.srt @@ -0,0 +1,39 @@ +1 +00:00:00,620 --> 00:00:03,310 +Now let's see if we are all on the same page + +2 +00:00:03,310 --> 00:00:07,180 +on the concept of testing specifications, and we're going to do that through + +3 +00:00:07,180 --> 00:00:09,280 +a quiz. What I would like for you to do is to + +4 +00:00:09,280 --> 00:00:12,830 +tell me, what are some possible test specifications that will satisfy the + +5 +00:00:12,830 --> 00:00:15,380 +requirements that we just saw? And I want you to express this + +6 +00:00:15,380 --> 00:00:17,270 +specification in terms of constraints on + +7 +00:00:17,270 --> 00:00:19,166 +the inputs. Which means, what constraints + +8 +00:00:19,166 --> 00:00:23,130 +should the input satisfy in order for this statement to be executed, + +9 +00:00:23,130 --> 00:00:26,210 +and this statement to be executed, and you can write your answer + +10 +00:00:26,210 --> 00:00:27,510 +here in these two slots. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/5 - Coverage Criteria Intro Quiz Solution - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/5 - Coverage Criteria Intro Quiz Solution - lang_en_vs4.srt new file mode 100644 index 0000000..2a1ff16 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/5 - Coverage Criteria Intro Quiz Solution - lang_en_vs4.srt @@ -0,0 +1,47 @@ +1 +00:00:00,080 --> 00:00:02,560 +To satisfy our first requirements, we need to find an + +2 +00:00:02,560 --> 00:00:06,300 +input that causes the execution of this statement. Because this statement + +3 +00:00:06,300 --> 00:00:09,080 +is executed only when result is greater than zero, our + +4 +00:00:09,080 --> 00:00:12,815 +test requirement is that a plus b must be greater than + +5 +00:00:12,815 --> 00:00:16,219 +0. When this is satisfied, this statement is executed therefore + +6 +00:00:16,219 --> 00:00:19,960 +any test case that implements this specification will cause the execution + +7 +00:00:19,960 --> 00:00:22,770 +of this statement. Similarly if we want to cover this statement + +8 +00:00:22,770 --> 00:00:25,220 +which is our second requirement we need to have a result + +9 +00:00:25,220 --> 00:00:27,370 +that is less than 0. Again, the result is equal + +10 +00:00:27,370 --> 00:00:29,490 +to a plus b, so all we need to do + +11 +00:00:29,490 --> 00:00:32,450 +is find the test case such that a plus b + +12 +00:00:32,450 --> 00:00:34,700 +is less than 0. So this is pretty straight forward. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/6 - Coverage Criteria Intro Quiz - lang_en_vs3.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/6 - Coverage Criteria Intro Quiz - lang_en_vs3.srt new file mode 100644 index 0000000..42b5720 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/6 - Coverage Criteria Intro Quiz - lang_en_vs3.srt @@ -0,0 +1,63 @@ +1 +00:00:00,220 --> 00:00:03,780 +So, now that we have our test specifications. Test specification number one that + +2 +00:00:03,780 --> 00:00:06,740 +says that a plus b must be greater than zero. And test specification + +3 +00:00:06,740 --> 00:00:09,990 +number two, for which a plus b must be less than zero. I'd + +4 +00:00:09,990 --> 00:00:12,580 +like to do another small quiz and ask you to write some test + +5 +00:00:12,580 --> 00:00:16,440 +cases that will implement these specifications. And I want you to write the + +6 +00:00:16,440 --> 00:00:19,780 +test cases in this format. I want you to specify for each one + +7 +00:00:19,780 --> 00:00:22,300 +of the test cases, what is the value of a that you need + +8 +00:00:22,300 --> 00:00:25,160 +to use. What is the value of b that you need to use. + +9 +00:00:25,160 --> 00:00:28,200 +And since a test case, I like to remind you, is + +10 +00:00:28,200 --> 00:00:30,960 +not just a set of inputs. But is the set of + +11 +00:00:30,960 --> 00:00:34,050 +inputs plus expected output? I also want you to specify, what + +12 +00:00:34,050 --> 00:00:37,040 +is the expected output for these test cases? And in particular, + +13 +00:00:37,040 --> 00:00:39,540 +since we have two characteristics of the output, one is the + +14 +00:00:39,540 --> 00:00:41,630 +color of the output and the other one is the actual + +15 +00:00:41,630 --> 00:00:44,040 +value. I want you to specify for each test case, what + +16 +00:00:44,040 --> 00:00:46,790 +is the expected output color and what is the expected value? diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/7 - Coverage Criteria Intro Quiz Solution - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/7 - Coverage Criteria Intro Quiz Solution - lang_en_vs4.srt new file mode 100644 index 0000000..749a57e --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/7 - Coverage Criteria Intro Quiz Solution - lang_en_vs4.srt @@ -0,0 +1,87 @@ +1 +00:00:00,090 --> 00:00:02,360 +In this case like in many other cases, there's not just a + +2 +00:00:02,360 --> 00:00:05,550 +single right answer because you can build many test cases that will + +3 +00:00:05,550 --> 00:00:09,730 +satisfy this test specification. So for example, we could pick value 3 + +4 +00:00:09,730 --> 00:00:13,640 +for a and value 9 for b. Those satisfy the specification because a + +5 +00:00:13,640 --> 00:00:16,400 +plus b is equal to 12 and therefore is greater than 0, + +6 +00:00:16,400 --> 00:00:19,470 +and therefore this is a test case that implements this test specification. + +7 +00:00:19,470 --> 00:00:21,960 +And in terms of results, what we expect to see is in + +8 +00:00:21,960 --> 00:00:25,200 +the case of a result greater than 0, the caller should be red, + +9 +00:00:25,200 --> 00:00:28,109 +and the upper value should be 12. And obviously for this test + +10 +00:00:28,109 --> 00:00:30,971 +specification, we just need to pick two inputs such that the sum + +11 +00:00:30,971 --> 00:00:33,674 +of the two inputs is less than 0. So for example, we + +12 +00:00:33,674 --> 00:00:37,125 +could pick minus 5 and minus 8. The output color in this + +13 +00:00:37,125 --> 00:00:40,230 +case is going to be blue, and the output value is going to be + +14 +00:00:40,230 --> 00:00:43,780 +minus 13. So, what we just saw is basically how we can + +15 +00:00:43,780 --> 00:00:47,060 +go from a piece of code to a set of requirements, which + +16 +00:00:47,060 --> 00:00:50,260 +are the interesting aspects of the code that we want to exercise. How we + +17 +00:00:50,260 --> 00:00:52,800 +can satisfy the requirements by finding + +18 +00:00:52,800 --> 00:00:54,700 +the right test specifications, and then how + +19 +00:00:54,700 --> 00:00:58,450 +we can initiate the test specifications into actual test cases. And this is + +20 +00:00:58,450 --> 00:01:01,010 +what we will do in general when doing white books testing. And we'll + +21 +00:01:01,010 --> 00:01:02,520 +do things likely different way, depending on + +22 +00:01:02,520 --> 00:01:04,230 +the specific criteria that we are considering. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/8 - Statement Coverage - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/8 - Statement Coverage - lang_en_vs4.srt new file mode 100644 index 0000000..23cfb0a --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/8 - Statement Coverage - lang_en_vs4.srt @@ -0,0 +1,291 @@ +1 +00:00:00,270 --> 00:00:03,080 +Now that we saw this overview of white-box testing, I'd like + +2 +00:00:03,080 --> 00:00:06,270 +to start talking about specific coverage criterion. And I'm going to start + +3 +00:00:06,270 --> 00:00:09,010 +with the first one, which is Statement Coverage. This criterion is + +4 +00:00:09,010 --> 00:00:12,355 +going to be characterized by two aspects, the first one is which + +5 +00:00:12,355 --> 00:00:15,580 +are the Test requirements for the criteria and the second one + +6 +00:00:15,580 --> 00:00:18,830 +is how we measure Coverage for that criteria. In the case + +7 +00:00:18,830 --> 00:00:22,200 +of statement coverage, these test requirements are all the statements in + +8 +00:00:22,200 --> 00:00:25,804 +the program. So this is the basic, the first the, the simplest + +9 +00:00:25,804 --> 00:00:29,860 +coverage criteria in the white-box arena. Let me remind you the assumption + +10 +00:00:29,860 --> 00:00:33,280 +that we made at the beginning. White-box testing is based on the assumption + +11 +00:00:33,280 --> 00:00:35,990 +that if there isn't a faulty element in the code, we need + +12 +00:00:35,990 --> 00:00:38,390 +to exercise it. We need to execute it, in order to find the + +13 +00:00:38,390 --> 00:00:41,260 +fault. And that's exactly what statement coverage does. If there is a + +14 +00:00:41,260 --> 00:00:44,450 +statement that is faulty in the code, we need to exercise it, in + +15 +00:00:44,450 --> 00:00:47,490 +order to find the fault. And therefore, a good measure of how well + +16 +00:00:47,490 --> 00:00:51,480 +we exercise the code, is the ratio of the number of executed statements. + +17 +00:00:51,480 --> 00:00:54,950 +So all the statements that my test cases executed, to the total + +18 +00:00:54,950 --> 00:00:58,570 +number of statements in the program. The higher this number, the better + +19 +00:00:58,570 --> 00:01:01,870 +I exercise my code. And we can also look at coverage criterion + +20 +00:01:01,870 --> 00:01:04,290 +in terms of questions. So what is the questions they were trying + +21 +00:01:04,290 --> 00:01:06,940 +to answer when we look at a specific set of test cases + +22 +00:01:06,940 --> 00:01:09,870 +and we assess the statement coverage that they achieved. And the question + +23 +00:01:09,870 --> 00:01:13,440 +is whether each statement in the program has been executed. So, statement + +24 +00:01:13,440 --> 00:01:16,920 +coverage is satisfied when all the statements in the program have been executed. + +25 +00:01:16,920 --> 00:01:19,640 +And we can satisfy to different degrees and the degrees to which it's + +26 +00:01:19,640 --> 00:01:23,320 +satisfied is measured by this value. So now let's go ahead and measure + +27 +00:01:23,320 --> 00:01:27,250 +statement coverage on our printSum example. What I'm going to show down here is + +28 +00:01:27,250 --> 00:01:30,500 +this progress bar in which we show the amount of coverage, the percentage of + +29 +00:01:30,500 --> 00:01:33,060 +coverage achieved. So what this means is that the, if I get to + +30 +00:01:33,060 --> 00:01:36,680 +this point I've covered 25% of the statements in the code. And my goal + +31 +00:01:36,680 --> 00:01:39,160 +is to get up here to cover all the statements in the code. + +32 +00:01:39,160 --> 00:01:42,000 +We have two test cases for this code. The first one that we just + +33 +00:01:42,000 --> 00:01:45,700 +saw, consists of the inputs a equal to 3 and b equal + +34 +00:01:45,700 --> 00:01:48,200 +to 9, and the second one has the inputs a is equal to + +35 +00:01:48,200 --> 00:01:50,840 +minus 5 and b is equal to minus 8. So now let's see + +36 +00:01:50,840 --> 00:01:53,450 +what happens when we run this test case. When we run this test + +37 +00:01:53,450 --> 00:01:56,920 +case, I'm going to show you by highlighting in the code the parts that + +38 +00:01:56,920 --> 00:02:00,430 +we cover when we start executing the code. We cover the first statement, + +39 +00:02:00,430 --> 00:02:01,910 +then we always execute the second + +40 +00:02:01,910 --> 00:02:04,180 +statement, which computes the result, we continue + +41 +00:02:04,180 --> 00:02:07,070 +the execution, we get to the if statement. If the result is greater + +42 +00:02:07,070 --> 00:02:10,038 +than zero, in this case our result is 12 because we + +43 +00:02:10,038 --> 00:02:12,360 +are working with the inputs 3 and 9, and therefore we + +44 +00:02:12,360 --> 00:02:15,710 +execute the true part of the if, we execute the statement. + +45 +00:02:15,710 --> 00:02:18,850 +And at this point, we just jump to the end. Because we + +46 +00:02:18,850 --> 00:02:21,530 +do not execute the else part of the statement, since we + +47 +00:02:21,530 --> 00:02:24,240 +have executed a true one, and therefore, we cover this final + +48 +00:02:24,240 --> 00:02:27,534 +statement. So at the end of the execution of this test + +49 +00:02:27,534 --> 00:02:32,313 +case, we cover one, two, three, four, five statement out of seven + +50 +00:02:32,313 --> 00:02:36,299 +which is roughly speaking 71%. So we can mark in here + +51 +00:02:36,299 --> 00:02:39,779 +that we more or less got to 71% of coverage for + +52 +00:02:39,779 --> 00:02:42,719 +this code. Now let's look at what happens when we execute + +53 +00:02:42,719 --> 00:02:45,700 +test case number two. In this case again, we execute the + +54 +00:02:45,700 --> 00:02:48,660 +first statement, the second statement, the third statement. In this case + +55 +00:02:48,660 --> 00:02:52,010 +though, the first statement, when it evaluates the value of result, + +56 +00:02:52,010 --> 00:02:54,250 +it sees that the result is not greater than zero because + +57 +00:02:54,250 --> 00:02:57,590 +our inputs are minus five and minus eight. Therefore, you will execute + +58 +00:02:57,590 --> 00:03:00,090 +line number five. And because the result is less than + +59 +00:03:00,090 --> 00:03:02,810 +zero, you will also execute line number six. So, at + +60 +00:03:02,810 --> 00:03:05,230 +this point, all of the statements in our code are + +61 +00:03:05,230 --> 00:03:09,360 +executed and therefore, we achieved a 100% statement coverage, which + +62 +00:03:09,360 --> 00:03:13,090 +was our goal. Before looking at other kinds of coverage, + +63 +00:03:13,090 --> 00:03:16,060 +let's see how our statement coverage is used in practice. + +64 +00:03:16,060 --> 00:03:19,170 +First of all, statement coverage is the most used kind + +65 +00:03:19,170 --> 00:03:22,830 +of coverage criterion in industry. Normally for company that uses + +66 +00:03:22,830 --> 00:03:27,200 +statement coverage, the typical coverage target is 80-90%, which + +67 +00:03:27,200 --> 00:03:29,690 +mean the outcome of the test should be such + +68 +00:03:29,690 --> 00:03:32,770 +that 80-90% of the statements are exercised at the + +69 +00:03:32,770 --> 00:03:34,880 +end of testing. So at this point, you might + +70 +00:03:34,880 --> 00:03:36,850 +be wondering, why don't we just shoot for 100%? + +71 +00:03:36,850 --> 00:03:38,580 +Why don't we try to cover all of the + +72 +00:03:38,580 --> 00:03:40,050 +code? We just saw that we could do it. + +73 +00:03:40,050 --> 00:03:41,730 +And so I'm going to ask you the same question. diff --git a/usth/ICT2.7/P4L3 White-Box Testing Subtitles/9 - Statement Coverage Quiz - lang_en_vs4.srt b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/9 - Statement Coverage Quiz - lang_en_vs4.srt new file mode 100644 index 0000000..893fe06 --- /dev/null +++ b/usth/ICT2.7/P4L3 White-Box Testing Subtitles/9 - Statement Coverage Quiz - lang_en_vs4.srt @@ -0,0 +1,15 @@ +1 +00:00:00,100 --> 00:00:01,540 +so, I would like to hear from you. Why + +2 +00:00:01,540 --> 00:00:03,850 +do you think that we don't aim normally at + +3 +00:00:03,850 --> 00:00:05,939 +100 % college but, slightly less than that and + +4 +00:00:05,939 --> 00:00:07,560 +I want you to put your answer right here. |