No | Text |
1 | Cem kaner, j.d., ph.d. |
2 | On the quality of qualitative measures |
3 | Appendix: more details on oracles |
No | Text |
1 | Software engineering professor and consumer advocate |
2 | Updating bbst to version 4.0 |
3 | Background: what is bbst |
4 | Updating to bbst 4.0: differences between the core bbst courses and domain testing |
5 | Differences between the core bbst courses and domain testing |
6 | Updating to bbst 4.0: learning objectives and structure of foundations 3.0 (2010) |
7 | Learning objectives and structure of foundations 3.0 (2010) |
8 | Updating to bbst 4.0: what should change |
9 | What we think should change |
10 | Updating to bbst 4.0: financial model & concluding thoughts |
11 | Financial model |
12 | Concluding thoughts |
13 | An update to bbst |
14 | Schools of software testing: a debate with rex black |
15 | Racial profiling in ferguson missouri? a note on statistical ınterpretation |
16 | Bbst domain testing pilot: a few seats still available (june 22-july 19) |
17 | On the quality of qualitative measures |
18 | On measurement |
19 | No, this is not qualitative |
20 | Ok, so what is qualitative? |
21 | Quality of qualitative |
22 | ın sum |
23 | References |
24 | Why propose an advanced certification in software testing? |
25 | ı used to see certifications as irrelevant (and misleading) |
26 | The market proved me wrong |
27 | One approach: open certification |
28 | Another alternative |
29 | A proposal for an advanced certification in software testing |
30 | The proposal |
31 | Administrative ıssues |
32 | Benefits |
33 | The current goal: a constructive discussion |
34 | New book: foundations of software testing–a bbst workbook |
35 | What’s in the book |
36 | Who the book is for |
37 | Evolution of the bbst courses |
38 | What is bbst™ ? |
39 | Today’s bbst courses |
40 | Background on the bbst course design |
41 | The collaboration with ast |
42 | An excerpt from our 2007 grant proposal |
43 | Thinking about the fraud against target |
44 | On the design of advanced courses in software testing |
45 | Content |
46 | ınst**utional considerations |
47 | Skill development |
48 | ınstructional style |
49 | Expectations of student performance |
50 | Credentialing |
51 | Last call for wtst 2014 |
52 | The “failure” of udacity |
53 | New book: the domain testing workbook |
54 | The 2014 workshop on teaching software testing is expanded |
55 | Teaching a new statistics course: ı need your recommendations |
56 | Credentialing in software testing: elaborating on my stpcon keynote |
57 | Transition |
58 | Decoupling ınstruction from credentialing |
59 | Dispersion of credentialing |
60 | ındustrial credentials |
61 | A few comments on our current state |
62 | Doma–time for a change |
63 | Presentation on software metrics |
64 | Follow-up testing in the bbst bug advocacy course |
65 | ınvestigating whether a minor-looking failure reflects a more serious bug |
66 | ınvestigating whether an obscure-looking failure will arise under more circ***stances |
67 | An overview of high volume automated testing |
68 | High-volume tests focused on ınputs |
69 | High-volume tests that exploit the availability of an oracle |
70 | High-volume tests that exploit the availability of existing tests or tools |
71 | A few more notes on history |
72 | ınteractive grading in university and pract**ioner cla***es: an experience report |
73 | What is ınteractive grading? |
74 | Costs and benefits of ınteractive grading |
75 | A more detailed report |
76 | A few more thoughts |
77 | Wtst 2013 call for partic****tion: teaching high volume automated testing (hivat) |
78 | Don’t censure people for disagreeing with us |
79 | Nominated for the “software test luminary” award |
80 | Bbst is now a registered trademark |
81 | ınstructor’s manual for the bbst courses |
82 | The oracle problem and the teaching of software testing |
83 | The oracle problem |
84 | The ınstructional problem |
85 | An ımportant ınstructional heuristic |
86 | Therefore |
87 | Oracles are necessarily ıncomplete |
88 | Oracles are heuristics |
89 | The bach / bolton consistency heuristics |
90 | Doug hoffman’s approach |
91 | Applying this to automation |
92 | An exam question |
93 | Cast 2012 metrics talk posted |
94 | Theses and dissertations about software testing |
No | Text |
1 | Structure of the core bbst courses |
2 | Structure of domain testing |
3 | 1. ınformation objectives drive the testing mission and strategy |
4 | 2. oracles are heuristic |
5 | 3. coverage is a multidimensional concept |
6 | 4. complete testing is impossible |
7 | 5. measurement is important, but difficult |
8 | (1) we published course workbooks for each of the three courses. the workbooks provide: |
9 | (2) we have overhauled the multiple-choice review questions in all three courses. |
10 | (3) we revised our model for instructor-student feedback. |
11 | ıf you are teaching or taking bbst, we suggest that you use the workbooks |
12 | Schools or strategies? |
13 | Unfortunate misbehavior |
14 | Examples of credibility-related considerations |
15 | Transferability |
16 | Dependability |
17 | Confirmability |
18 | Authorization by the applicant |
19 | Education (academic) |
20 | Education (practical) |
21 | Examination |
22 | Professional achievement |
23 | References |
24 | Professional experience |
25 | Continuing education |
26 | Code of ethics |
27 | All the course slides |
28 | A transcript of the six lectures. |
29 | Four orientation activities |
30 | Two application activities |
31 | Advice on answering our essay-style exam questions |
32 | Author reflections |
33 | 3. our current course (black box software testing—bbst) |
34 | 3.1 we organize cla***es around learning units that typically include: |
35 | Excerpt from the proposal’s references: |
36 | 1. what are the characteristics of a genuinely “advanced” testing course? |
37 | 2. what should the characteristics be for an advanced certification in software testing? |
38 | Background of the students |
39 | My approach to the course |
40 | Examples wanted |
41 | Thanks for your help! |
42 | ınstruction |
43 | Credentialing |
44 | ıstqb |
45 | The old boys’ club |
46 | Miagi-do |
47 | Other ıdeas on the horizon |
48 | 1. vary your behavior |
49 | 2. vary the options and settings of the program |
50 | 3. vary data that you load into the program |
51 | 4. vary the software and hardware environment |
52 | 1. uncorner your corner cases |
53 | 2. look for configuration dependence |
54 | 3. check whether the bug is new to this version |
55 | 4. check whether bugs like this one already appear in the bug database |
56 | 5. check whether bugs of this kind appear in other programs |
57 | High-volume parametric variation |
58 | High-volume combination testing |
59 | ınput fuzzing |
60 | Hostile data stream testing |
61 | Function equivalence testing |
62 | Constraint checks |
63 | ınverse operations |
64 | State-model based testing (smbt) |
65 | Diagnostics-based testing |
66 | Long-sequence regression testing (lsrt) |
67 | High-volume protocol testing |
68 | Load-enhanced functional testing |
69 | Costs |
70 | Benefits for the students |
71 | Benefits for me |
72 | Structural/logistical matters |
73 | Example 1: midterm exams |
74 | Example 2: practical a***ignments |
75 | Example 3: research essays |
76 | What are these doc***ents? |
77 | Why ı recommend them |
78 | What’s in the bibliography |
79 | Acknowledgments |
No | Text |
1 | Critique: (5) |
2 | Tentative decisions: (5) |
3 | A supplementary video: |
4 | Which exams are suitable and which are advanced? |
5 | What about the reliability and the validity of the exams? |
6 | The bug a***ignment |
7 | The domain testing analysis |
8 | Archives |
9 | Categories |
10 | Meta |
No | Text |
1 | We’re looking for comments, including criticism and alternative suggestions: |
2 | We’re looking for recommendation-worthy readings |
3 | Course description here (4) |
4 | Courseware available here (3) |
5 | Course Workbook (2) |
6 | Course Textbook |
7 | Orientation activities (2) |
8 | Application activities (2) |
9 | Multiple-choice quizzes (2) |
10 | Various other discussions (2) |
11 | An essay-style final exam (2) |
12 | Interactive Grading |
13 | Focus On One Technique |
14 | The workbook is a textbook |
15 | Application Videos |
16 | This is the primary instructional innovation in this course, and students have praised it highly. |
17 | Capstone Project Instead of an Exam: |
18 | Information objectives drive the testing mission and strategy |
19 | Oracles are heuristic |
20 | Coverage is a multidimensional concept |
21 | Complete testing is impossible |
22 | Measurement is important, but hard |
23 | Help students develop learning skills that are effective in the online-course context. |
24 | Foster the att**ude that our a***essments are helpfully hard without being risky |
25 | Most |
26 | Please send us good pointers to articles / blog posts that Foundations students can read to learn more about these questions (and ways to answer them). |
27 | a***essment |
28 | Focus |
29 | If you are teaching a BBST course, we strongly recommend that you use the course Workbook as a course text. It changes the student experience, and the feedback we continually get is that the change is strongly positive. |
30 | as individuals, we get to choose |
31 | However, if you look at the actual numbers from Ferguson, it is not clear to me that the conclusions of racial profiling, conclusions like the ones quoted above, that have appeared in every news source that I respect, are justified by the data. |
32 | The numbers don’t tell you what they mean. Even the most statistically significant trends must be interpreted by people. |
33 | Whenever possible, you should look at your data in many ways, to see if they tell you the same story. If they don’t, you need to investigate further, and maybe fix your model. |
34 | A few numbers in isolation tell you very little, often much less than you would initially imagine |
35 | If you design your research (or your management) so that you will see only a few numbers at the end, you are designing tunnel vision into your work. You are creating your own context for bad interpretations and bad decisions |
36 | . |
37 | BBST Domain Testing |
38 | “All quant**ative data is based upon qualitative judgments.” |
39 | If you think your qualitative measurement methods are easier, faster and cheaper than the quant**ative alternatives, you are probably not doing the qualitative work very well. |
40 | Did you collect the data in a reasonable way? |
41 | How much detail?: |
42 | Prolonged engagement |
43 | Persistent observation |
44 | Triangulation and convergence. |
45 | Are you summarizing the data fairly? |
46 | How are you managing your biases (people are often not conscious of the effects of their biases) as you select and organize your observations? |
47 | Are you prone to wishful thinking or to trying to please (or displease) people in power? |
48 | Peer debriefing |
49 | Disconfirming case analysis |
50 | Progressive subjectivity |
51 | Member checks |
52 | reasonably attainable (2) |
53 | credible (2) |
54 | scalable (2) |
55 | commercially viable (2) |
56 | Self-studiers. |
57 | In-house trainers. |
58 | Students in instructor-led courses. |
59 | Buy the Book |
60 | knowledge |
61 | cognitive processing |
62 | Knowledge dimension |
63 | Cognitive Process |
64 | Video lecture and lecture slides |
65 | Application to a product under test. |
66 | Cla***room activities. |
67 | Examples. |
68 | a***igned readings. |
69 | a***ignments, |
70 | Study guide questions. |
71 | substance |
72 | state and enforce prerequisites |
73 | accept transfer credit |
74 | sets expectations |
75 | Ultimately, we concluded that teaching through exercises is still our best shot at helping people develop skill, but that we needed to provide a conceptual structure for the exercises that could give students a strategy for approaching new problems. We created a schema—an 18-step cognitive structure that describes how we do a domain analysis—and we present every exercise and every example in the context of that schema. |
76 | notional variable |
77 | as if |
78 | software testing |
79 | Industrial credentialing will probably get more important, not less important, over the next 20 years. Rather than wasting everyone’s time whining about the shortcomings of current credentials, do the work needed to create a viable alternative. |
80 | Credentials based on what you know, what you can do, or what you have actually done are a lot more egalitarian than those based on who says they respect you. |
81 | Some people have asked me why I posted this graphic. It is a token of solidarity with the Human Rights Campaign. This is one of several images that HRC supporters posted on their websites while the United States Supreme Court was hearing arguments about the Defense of Marriage Act. |
82 | more serious |
83 | 1. Vary your behavior. |
84 | Summary |
85 | Summary: |
86 | When I do interactive grading |
87 | When I explicitly build interactive grading into my cla***: |
88 | WTST CALL FOR PARTIc****TION |
89 | TEACHING HIGH VOLUME AUTOMATED TESTING (HiVAT) |
90 | BACKGROUND ON THE WORKSHOP TOPIC |
91 | TO ATTEND AS A PRESENTER |
92 | TO ATTEND AS A NON-PRESENTING PARTIc****NT: |
93 | HOW THE MEETING WILL WORK |
94 | LOCATION AND TRAVEL INFORMATION |
95 | Airport |
96 | Hotel |
97 | a software testing oracle is a tool that helps you decide whether the program pa***ed your test |
98 | do find them helpful for evaluating test results |
99 | don’t find them helpful for designing tests. |
100 | worthless for designing automated tests |
101 | In my courses, the consistencies capture my students’ imagination and interfere with their thinking about oracles that would be useful for designing tests, especially automated tests. |
102 | incorrectly decide that the program pa***ed the test because its outputs matched the expected outputs |
103 | incorrectly decide that the program failed the test because its outputs did not match the expected results |
104 | Note: Don’t just echo back a consistency heuristic. Be specific in your description of a relevant oracle and of the types of information or bugs that you expect. |
No | Text |
1 | SPLAT! Requirements bugs on the information superhighway |
No | Text |
1 | We’re looking for comments, including criticism and alternative suggestions: |
2 | We’re looking for recommendation-worthy readings |
3 | Evolution of the BBST Courses |
4 | If you are already familiar with BBST, skip this (4) |
5 | domain testing |
6 | doing the technique well |
7 | application videos |
8 | Orientation activities (2) |
9 | Application activities (2) |
10 | Multiple-choice quizzes (2) |
11 | Various other discussions (2) |
12 | An essay-style final exam (2) |
13 | Interactive Grading (2) |
14 | Focus On One Technique |
15 | The workbook is a textbook |
16 | Application Videos |
17 | This is the primary instructional innovation in this course, and students have praised it highly. |
18 | Capstone Project Instead of an Exam: |
19 | content |
20 | skills |
21 | att**udes |
22 | software testing (2) |
23 | Java programming |
24 | communication skills |
25 | Foundations 2.0 and 3.0 |
26 | Information objectives drive the testing mission and strategy |
27 | Oracles are heuristic |
28 | Coverage is a multidimensional concept |
29 | Complete testing is impossible |
30 | Measurement is important, but hard |
31 | Help students develop learning skills that are effective in the online-course context. |
32 | Foster the att**ude that our a***essments are helpfully hard without being risky |
33 | for students who make an honest effort |
34 | Bug Advocacy (3) |
35 | Test Design (4) |
36 | Foundations (9) |
37 | Most |
38 | do |
39 | other (2) |
40 | not (3) |
41 | information objective (2) |
42 | mission (4) |
43 | The distinction between information objective and mission is too fine-grained |
44 | The presentation of context got buried in a ma*** of other details |
45 | We must present more contexts and/or characterize them more explicitly |
46 | information objectives |
47 | strategy. |
48 | usually |
49 | The terminology created at least as much confusion as insight. |
50 | The terminology is redundant and uninformative. |
51 | heuristic |
52 | Heuristic is a magic word. |
53 | fallible |
54 | incomplete. |
55 | The nature of incompleteness |
56 | pa***ed |
57 | Human observers don’t eliminate this incompleteness |
58 | can (2) |
59 | I don’t think you achieve effective oracle diversity by choosing to “explore” |
60 | How can we compensate for inevitable incompleteness, (a) when we work with automated test execution and evaluation and (b) when we work with human observers? |
61 | There are two uses of oracles. We emphasized the one that is wrong for Foundations |
62 | The test-design oracle: |
63 | The tester-expectations oracle: |
64 | What are you looking for with this (test or) series of tests |
65 | What are these tests blind to? |
66 | How will you look for those other potential problems? |
67 | Can you detect these potential problems with code? |
68 | If not, can you train people to look for them? |
69 | Which problems are low-priority enough, or low-likelihood enough, that you should choose to run general exploratory tests and rely on luck and hope to help you stumble across them if they are there? |
70 | today |
71 | Please send us good pointers to articles / blog posts that Foundations students can read to learn more about these questions (and ways to answer them). |
72 | few (2) |
73 | how many? |
74 | measure |
75 | how many different rodents do you need to test? |
76 | all (2) |
77 | encourage |
78 | bonus |
79 | won’t require |
80 | little |
81 | bad |
82 | enormous |
83 | a***essment |
84 | Focus |
85 | http://bbst.info. Foundations, |
86 | Bug Advocacy, |
87 | some (7) |
88 | interactive grading |
89 | If you are teaching a BBST course, we strongly recommend that you use the course Workbook as a course text. It changes the student experience, and the feedback we continually get is that the change is strongly positive. |
90 | Lessons Learned in Software Testing |
91 | BBST-Domain Testing. |
92 | strategy |
93 | divisions into schools |
94 | also (2) |
95 | as individuals, we get to choose |
96 | choose (3) |
97 | and in my view, tarnish the Context-Driven Testing brand less |
98 | However, if you look at the actual numbers from Ferguson, it is not clear to me that the conclusions of racial profiling, conclusions like the ones quoted above, that have appeared in every news source that I respect, are justified by the data. |
99 | supposed to (3) |
100 | must |
101 | didn’t |
102 | these |
103 | The numbers don’t tell you what they mean. Even the most statistically significant trends must be interpreted by people. |
104 | Whenever possible, you should look at your data in many ways, to see if they tell you the same story. If they don’t, you need to investigate further, and maybe fix your model. |
105 | A few numbers in isolation tell you very little, often much less than you would initially imagine |
106 | If you design your research (or your management) so that you will see only a few numbers at the end, you are designing tunnel vision into your work. You are creating your own context for bad interpretations and bad decisions |
107 | BBST Domain Testing |
108 | (Sorry, the cla*** is now full and we have had to close registration CK — 5/23/2014) |
109 | The Domain Testing Workbook (3) |
110 | Conference of the a***ociation for Software Testing (2) |
111 | there are problems with the application of traditional metrics in our field |
112 | From quantification to methodolatry |
113 | “All quant**ative data is based upon qualitative judgments.” |
114 | might sometimes |
115 | are problems |
116 | subjective |
117 | subject to |
118 | bias at every level (how the data are gathered or selected, stored, analyzed, interpreted and reported). |
119 | selected |
120 | how much will this cost? |
121 | lot |
122 | There is trouble here |
123 | Things seem OK, nothing particularly good or bad to report now. |
124 | Things go well. |
125 | If you think your qualitative measurement methods are easier, faster and cheaper than the quant**ative alternatives, you are probably not doing the qualitative work very well. |
126 | Why should someone else trust your work? |
127 | Did you collect the data in a reasonable way? |
128 | Coding |
129 | Triangulation |
130 | degree of convergence |
131 | Are you summarizing the data fairly? |
132 | How are you managing your biases (people are often not conscious of the effects of their biases) as you select and organize your observations? |
133 | Are you prone to wishful thinking or to trying to please (or displease) people in power? |
134 | does |
135 | you |
136 | your work |
137 | Your attention to methodology and fairness reflect on your character and trustworthiness. |
138 | reliability |
139 | ability to be confirmed |
140 | actually |
141 | your data |
142 | are (4) |
143 | International Journal of Qualitative Methods |
144 | Measuring and Managing Performance in Organizations |
145 | The Leprechauns of Software Engineering: How folklore turns into fact and what to do about it. |
146 | Qualitative Inquiry and Research Design: Choosing Among Five Approaches |
147 | Constructing the Subject: Historical Origins of Psychological Research |
148 | . (3) |
149 | The SAGE Handbook of Qualitative Research |
150 | Doing Naturalistic Inquiry: A Guide to Methods |
151 | “In transition”: An activity theoretical analysis examining electronic portfolio tools’ mediation of the preservice teacher’s authoring experience. |
152 | Society for Information Technology and Teacher Education |
153 | British Journal of Occupational Therapy. |
154 | Fourth Generation Evaluation. |
155 | Pacific Northwest Software Quality Conference (2) |
156 | Auditory and visual synchronization performance over long and short intervals |
157 | Software Testing & Quality Engineering, |
158 | 15th International Software Quality Conference (Quality Week), |
159 | IFIP Working Group 10.4 meeting on Software Dependability |
160 | Software Test Professionals Conference |
161 | 86th Annual Convention of the American Psychological a***ociation |
162 | Qualitative Research & Evaluation Methods |
163 | Experimental and Quasi-Experimental Designs for Generalized Causal Inference |
164 | must be |
165 | and the names of the people we tie those definitions to? |
166 | clearly better |
167 | yet |
168 | and |
169 | as are the other certifications in our field |
170 | as can the others |
171 | but the others have security risks too |
172 | neither do the others (2) |
173 | and neither do the others |
174 | We would resolve this by requiring the applicant for certification to explain in writing how and why her or his education has proved to be relevant to her or his experiences as a tester and why it should be seen as relevant education for someone in the field. |
175 | advanced (2) |
176 | Someone has to decide which courses are suitable and which are advanced. |
177 | Someone has to decide which exams are suitable and which are advanced. |
178 | not yet |
179 | reasonably attainable (2) |
180 | credible (2) |
181 | scalable (2) |
182 | commercially viable (2) |
183 | in principle |
184 | reflections (2) |
185 | Self-studiers. |
186 | and provides detailed feedback for all of the orientation activities, and provides several design notes on the orientation and application activities. |
187 | In-house trainers. |
188 | Students in instructor-led courses. |
189 | Foundations of Software Testing: A BBST Workbook |
190 | BBST:Domain Testing |
191 | Foundations of Software Testing |
192 | Testing Circus |
193 | Building a free courseware community around an online software testing curriculum |
194 | very complex idea. |
195 | too complex |
196 | Adaptation and Implementation of an Activity-Based Online or Hybrid Course in Software Testing |
197 | knowledge |
198 | cognitive processing |
199 | Knowledge dimension |
200 | Factual Knowledge |
201 | Conceptual Knowledge |
202 | Procedural Knowledge |
203 | Metacognitive Knowledge |
204 | Cognitive Process |
205 | Remembering |
206 | Understanding |
207 | Applying |
208 | Analyzing |
209 | Evaluating |
210 | Creating |
211 | Video lecture and lecture slides |
212 | lecturer’s |
213 | Stored lectures |
214 | Application to a product under test. |
215 | Cla***room activities. |
216 | Examples. |
217 | Schaum’s Outline |
218 | Improving the Education of Software Testers (2) |
219 | a***igned readings. |
220 | a***ignments, |
221 | Study guide questions. |
222 | A Taxonomy for Learning, Teaching & a***essing: A Revision of Bloom’s Taxonomy of Educational Objectives |
223 | What’s the Use of Lectures? |
224 | Taxonomy of Educational Objectives: Book 1 Cognitive Domain |
225 | Cognition and categorization |
226 | e-Learning and the Science of Instruction |
227 | GVU Technical Report GVU-05-30 |
228 | CHI ’06 (Extended Abstracts on Human Factors in Computing Systems) |
229 | International Conference of Computers in Education |
230 | GVU Technical Report GVU-04-18 |
231 | IEEE Transactions on Education |
232 | 49 |
233 | Engineering Education |
234 | 78 |
235 | Proceedings of the 5th annual SIGCSE/SIGCUE ITiCSE Conference on Innovation and technology in computer science education |
236 | The Professor’s Guide to Teaching: Psychological Principles and Practices |
237 | The Cognitive Psychology of School Learning |
238 | International Journal of Qualitative Studies in Education |
239 | 12 |
240 | Transfer of learning: Cognition, instruction, and reasoning |
241 | Workshop on the Teaching of Software Testing (WTST) |
242 | Conference on Software Engineering Education & Training |
243 | Teaching of Psychology |
244 | 28 |
245 | Situated Learning: Legitimate Peripheral Partic****tion |
246 | a***essment of authentic performance in school mathematics |
247 | Journal of Experimental Psychology: Applied |
248 | 8 |
249 | Electronic Journal of Science Education |
250 | Psychological Review |
251 | 85 |
252 | Department of Computer Sciences |
253 | How Students Learn: Reforming Schools Through Learner-Centered Education |
254 | Journal of Asynchronous Learning Networks |
255 | 3 |
256 | Handbook of Distance Education |
257 | Canadian Journal of Psychology |
258 | 59 |
259 | Problems in probability theory, mathematical statistics and theory of random functions |
260 | Reading Psychology |
261 | 21 |
262 | Training complex cognitive skills: A four-component instructional design model for technical training |
263 | American Educational Research Journal |
264 | 14 |
265 | two |
266 | substance |
267 | state and enforce prerequisites |
268 | accept transfer credit |
269 | sets expectations |
270 | skill |
271 | scaffolding |
272 | Domain Testing (3) |
273 | should (4) |
274 | The problem with this is that it a***umes that teaching = lectures. For most students, this is not true. Students learn by doing things and getting feedback. By writing essays and getting feedback. By writing code and getting feedback. By designing tests and getting feedback. The student activities—running them, coaching students through them, critiquing student work, suggesting follow-up activities for individuals to try next—do not easily scale. I spent about 15 hours this week in face-to-face meetings with individual students, coaching them on statistical analysis or software testing. Next week I will spend about 15 hours in face-to-face meetings with local students or Skype sessions with online students. This is hard work for me, but my students tell me they learn a lot from this. |
275 | It is this argument that fails when 95% of the students flunk out or drop out. But I think it fails worse when you consider what these students are learning. How hard are the tests they are taking or the a***ignments they are submitting? How carefully graded is the work—not just how accurate is the grading, though that can certainly be a big issue with computerized grading—but also, how informative is the feedback from grading? |
276 | their |
277 | Students learn a lot from each other. Back when I paid attention to the instructional-research literature, I used to read studies that reported graduating students saying they learned more from each other than from the professors. |
278 | much more |
279 | The Domain Testing Workbook. |
280 | Equivalence Cla*** Analysis |
281 | Boundary Testing |
282 | transfer problem |
283 | transfer |
284 | Ultimately, we concluded that teaching through exercises is still our best shot at helping people develop skill, but that we needed to provide a conceptual structure for the exercises that could give students a strategy for approaching new problems. We created a schema—an 18-step cognitive structure that describes how we do a domain analysis—and we present every exercise and every example in the context of that schema. |
285 | Testing Computer Software |
286 | We were sooooooo wrong. |
287 | Domain Testing Schema |
288 | us |
289 | notional variable (2) |
290 | Thanks for your help! |
291 | many |
292 | Instruction |
293 | Credentials |
294 | Anyone |
295 | Industrial credentialing will probably get more important, not less important, over the next 20 years. Rather than wasting everyone’s time whining about the shortcomings of current credentials, do the work needed to create a viable alternative. |
296 | Lessons Learned |
297 | good (2) |
298 | only (2) |
299 | Open Certification, |
300 | Credentials based on what you know, what you can do, or what you have actually done are a lot more egalitarian than those based on who says they respect you. |
301 | any (3) |
302 | Some people have asked me why I posted this graphic. It is a token of solidarity with the Human Rights Campaign. This is one of several images that HRC supporters posted on their websites while the United States Supreme Court was hearing arguments about the Defense of Marriage Act. |
303 | more serious |
304 | Testing Computer Software. |
305 | What happens if I do |
306 | instead of |
307 | ? |
308 | persistent variables |
309 | that (3) |
310 | Summary |
311 | really |
312 | manual (2) |
313 | automated (2) |
314 | more (2) |
315 | less |
316 | point |
317 | all-pairs |
318 | all-triples |
319 | cause-effect graphing |
320 | This tests all the combinations but the set might be impossibly large. |
321 | Generate random values for the inputs, stopping when some large number have been tested. |
322 | Use an algorithm that optimizes the set of combinations in some way. As a very simple example, if you are going to test a million combinations, you could divide the space of combinations into a million same-size, non-overlapping subsets and sample one value of each. Or you could use a sequential algorithm that a***igns values for the next combination by creating a test that is distant (according to some distance function) from all previous tests. |
323 | everything that might possibly be interesting |
324 | some types of failures |
325 | If Calc does it the same way as Excel, that’s good enough, even if Excel makes a few calculation errors. |
326 | constraint (2) |
327 | lots of |
328 | pick only tests that the program pa***es. |
329 | Summary: |
330 | It wasn’t until I tried it that |
331 | my personal experiences and reflections |
332 | When I do interactive grading |
333 | before |
334 | for the first time |
335 | When I explicitly build interactive grading into my cla***: |
336 | feels like |
337 | feels as though |
338 | lot of |
339 | feel |
340 | What were you thinking? |
341 | What do you think this word in the question means? If I gave you a different explanation of what this word means, how would that affect your answer to the question? |
342 | Give me an example of what you are describing? |
343 | Can you give me a real-life example of what you are describing? For example, suppose we were working with OpenOffice. How would this come up in that project? |
344 | Can you explain this with a diagram? Show me on my whiteboard. |
345 | How would you answer this if I changed the question’s wording this way? |
346 | How would someone actually do that? |
347 | Why would anyone want to do that task that way? Isn’t there a simpler way to do the same thing? |
348 | weak competence |
349 | weak performance |
350 | no competence |
351 | poor performance |
352 | writes poorly or in a disorganized way |
353 | misunderstands the question. |
354 | competence |
355 | performance |
356 | Benefits for Me |
357 | their actual words on the exam |
358 | Show me what makes this a ‘B’ |
359 | I ask them to justify the higher grade by showing how their answer provides the information that the grading guide says is required or creditable for this question. |
360 | I explain that I can only grade what they say, not what they don’t say but I think they know anyway. |
361 | I often give them additional credit, but we’ll talk about why it is that they missed writing down that part of the answer, and how they would structure an answer to a question like this in the future so that their performance on the next exam is better. |
362 | Usually this doesn’t work, but sometimes I give strong points for an analysis that is different from mine but justifiable. I might add their analysis to the grading guide as another path to high points for that answer. |
363 | I try to lead from here to a different discussion–How did you miss this? What is the hole in your study strategy? |
364 | I might say to the student, “That’s a good analysis. Your answer on paper was worth 3/10. I’m going to record a 7/10 to reflect how much better a job you can actually do, but next time we won’t have interactive grading so you’ll have to show me this quality in what you write, not in the meeting. If you give an answer this good on the next exam, you’ll get a 9 or 10. |
365 | long |
366 | Here’s another case where you answered only 2 of the 3 parts of the question. I think you need to make a habit of writing an outline of your answer, check that the outline covers every part of the question, and then fill in the outline. |
367 | tells me |
368 | constructive |
369 | unconfirmed |
370 | might |
371 | invalid |
372 | did |
373 | tomorrow |
374 | That’s |
375 | this (2) |
376 | oracle (2) |
377 | reference program (2) |
378 | Oracle |
379 | miss |
380 | false alarm |
381 | a software testing oracle is a tool that helps you decide whether the program pa***ed your test |
382 | problem |
383 | feels |
384 | why |
385 | do find them helpful for evaluating test results |
386 | don’t find them helpful for designing tests. |
387 | worthless for designing automated tests |
388 | In my courses, the consistencies capture my students’ imagination and interfere with their thinking about oracles that would be useful for designing tests, especially automated tests. |
389 | as part of the exam question itself |
390 | a few |
391 | the problem is in the students |
392 | a lot of |
393 | the problem is in the instruction |
394 | something |
395 | expected results |
396 | On Testing Nontestable Programs |
397 | partial oracles |
398 | incorrectly decide that the program pa***ed the test because its outputs matched the expected outputs |
399 | incorrectly decide that the program failed the test because its outputs did not match the expected results |
400 | check |
401 | Heisenbug |
402 | What is it that makes the tester decide this is wrong behavior? |
403 | obvious |
404 | Cognition & Categorization |
405 | as |
406 | regression |
407 | self-verifying data |
408 | physical model |
409 | business model |
410 | statistical model |
411 | statistical oracle |
412 | state model |
413 | interaction model |
414 | calculation oracles |
415 | inverse oracle |
416 | Note: Don’t just echo back a consistency heuristic. Be specific in your description of a relevant oracle and of the types of information or bugs that you expect. |
417 | Theses |
418 | dissertations. |
419 | student work |
Icon | Domain | IP | Alexa Rank |
Toriavey.com | 192.124.249.158 | 52.834 | |
Increvenue.com | 192.124.249.158 | 143.853 | |
Tisserand.com | 192.124.249.158 | 438.920 | |
Worldpoliticsreview.com | 192.124.249.158 | 148.896 |
Icon | Domain | IP | Alexa Rank |
5Bestthings.com | 192.124.249.10 | 81.109 | |
Accurateshooter.com | 192.124.249.10 | 209.884 | |
Seekvectorlogo.com | 192.124.249.10 | 94.441 | |
Microland.com | 192.124.249.10 | 254.831 | |
Pynative.com | 192.124.249.10 | 29.538 | |
Purplemath.com | 192.124.249.10 | 18.660 | |
Storybrand.com | 192.124.249.10 | 41.335 | |
Georgiancollege.ca | 192.124.249.10 | 59.251 | |
Amoca.org | 192.124.249.10 | 3.008.188 | |
Pattayaunlimited.com | 192.124.249.10 | 966.789 | |
Iwmagazine.com | 192.124.249.10 | 2.558.408 | |
Tfltruck.com | 192.124.249.10 | 188.526 | |
Somaliaonline.com | 192.124.249.10 | 862.682 | |
Pymnts.com | 192.124.249.10 | 9.107 | |
Topdawgmovers.com | 192.124.249.10 | - | |
Vas***electricals.com | 192.124.249.102 | 109.878 | |
Braintraining4Dogs.com | 192.124.249.102 | 55.640 | |
Catster.com | 192.124.249.102 | 90.877 | |
Dcmshriram.com | 192.124.249.102 | 577.727 | |
4Hosea.org | 192.124.249.102 | 3.603.702 | |
Phonicarecords.com | 192.124.249.103 | 458.720 | |
Lawctopus.com | 192.124.249.103 | 55.504 | |
Lollipop.com | 192.124.249.103 | 7.518.653 | |
Mica.ac.in | 192.124.249.104 | 150.910 | |
Sunstar.com.ph | 192.124.249.104 | 43.988 | |
Craftsman-Book.com | 192.124.249.104 | 357.415 | |
Mommysmemorandum.com | 192.124.249.104 | 86.690 | |
Resco.net | 192.124.249.104 | 486.140 | |
Blumint.co | 192.124.249.104 | 559.643 | |
Theblendergirl.com | 192.124.249.105 | 611.981 | |
Cookrecorder.com | 192.124.249.105 | 671.294 | |
Sourcemore.com | 192.124.249.105 | 91.288 | |
Themediterraneandish.com | 192.124.249.105 | 53.265 | |
Webtoffee.com | 192.124.249.105 | 39.435 | |
Bergerbullets.com | 192.124.249.105 | 162.468 | |
Sscc.edu | 192.124.249.105 | 371.066 | |
Powherful.org | 192.124.249.105 | 4.896.519 | |
Incinolet.com | 192.124.249.105 | 965.393 | |
Urban-Drinks.de | 192.124.249.106 | 259.870 | |
Ft86Speedfactory.com | 192.124.249.106 | 111.479 | |
Bluealphabelts.com | 192.124.249.106 | 874.727 | |
Southcharlestonwv.org | 192.124.249.106 | - | |
Spectorshockey.net | 192.124.249.106 | 2.047.646 | |
Americansti.org | 192.124.249.106 | - | |
Bestfitbybrazil.com | 192.124.249.107 | 194.918 | |
Onlineadvertisingschool.com | 192.124.249.107 | 233.270 | |
Gettingtotruelove.com | 192.124.249.107 | 247.093 | |
Loavesanddishes.net | 192.124.249.107 | 124.681 | |
Positronx.io | 192.124.249.107 | 37.925 | |
Allontario.ca | 192.124.249.107 | 371.207 | |
Zoolert.com | 192.124.249.108 | 67.388 | |
Pushalert.co | 192.124.249.108 | 31.561 | |
Revelstokemountaineer.com | 192.124.249.108 | 1.347.940 | |
Websterjournal.com | 192.124.249.108 | 2.847.855 | |
Honestlywtf.com | 192.124.249.109 | 154.475 | |
Saco-Ksa.com | 192.124.249.109 | 31.723 | |
Askaprepper.com | 192.124.249.109 | 179.681 | |
Vintagekitchennotes.com | 192.124.249.109 | 88.078 | |
Joomdonation.com | 192.124.249.11 | 118.492 | |
Cla***lawgroup.com | 192.124.249.11 | 572.404 | |
Maryknollsisters.org | 192.124.249.11 | - | |
Omicron.com | 192.124.249.11 | 1.249.989 | |
3Axis.co | 192.124.249.110 | 24.796 | |
Cbu.ac.zm | 192.124.249.110 | 149.206 | |
Gokorea.co.uk | 192.124.249.110 | 1.600.974 | |
Ioepa.com.br | 192.124.249.110 | 292.805 | |
Panningtheglobe.com | 192.124.249.110 | 92.095 | |
Alqasba.ae | 192.124.249.110 | 1.342.150 | |
Grandcanyoncvb.org | 192.124.249.110 | 3.954.718 | |
Stacyknows.com | 192.124.249.111 | 327.044 | |
Saludtools.com | 192.124.249.111 | 659.977 | |
Trustmerchantbank.com | 192.124.249.111 | 1.067.998 | |
Sftt.org | 192.124.249.111 | - | |
Hondaphil.com | 192.124.249.111 | 445.967 | |
Businessworld.ie | 192.124.249.111 | 1.210.537 | |
Carmelitenuns.uk | 192.124.249.111 | 6.641.382 | |
Techbullion.com | 192.124.249.111 | 24.491 | |
Cprprofessor.com | 192.124.249.111 | - | |
Tubefilter.com | 192.124.249.112 | 49.649 | |
Thekingcenter.org | 192.124.249.112 | 359.167 | |
Soberjulie.com | 192.124.249.112 | 1.060.121 | |
Josemariasison.org | 192.124.249.112 | 6.362.956 | |
Newcreation.org.sg | 192.124.249.113 | 154.124 | |
Riversidecountyit.org | 192.124.249.113 | - | |
Thingmagic.com | 192.124.249.113 | - | |
Writeaprisoner.com | 192.124.249.114 | 108.088 | |
Cliver.to | 192.124.249.114 | 19.270 | |
Dqindia.com | 192.124.249.114 | 51.967 | |
Clubbcanada.com | 192.124.249.114 | 1.300.264 | |
Aaawt.com | 192.124.249.114 | 9.268.457 | |
Paymentsjournal.com | 192.124.249.115 | 101.622 | |
Marineinsight.com | 192.124.249.115 | 39.995 | |
Patpat.lk | 192.124.249.115 | 132.577 | |
Leobaecktemple.org | 192.124.249.115 | 3.756.820 | |
Lexonlimo.com | 192.124.249.115 | - | |
Measuringu.com | 192.124.249.117 | 151.883 | |
Hyperice.com | 192.124.249.117 | 32.887 | |
Speakersponsor.com | 192.124.249.117 | 136.751 | |
Indiesponsor.com | 192.124.249.117 | 136.086 | |
Eventsincolorado.com | 192.124.249.117 | 159.027 | |
Belleofthekitchen.com | 192.124.249.117 | 168.815 | |
Thecatalinaislander.com | 192.124.249.117 | 2.850.513 | |
Taylorwoodrow.com | 192.124.249.117 | - | |
Berks***rehathaway.com | 192.124.249.118 | 38.784 | |
Fandffirearms.com | 192.124.249.118 | 718.843 | |
Jamminjava.com | 192.124.249.118 | - | |
Jamesharrisgallery.com | 192.124.249.118 | - | |
c***minsengines.com | 192.124.249.118 | - | |
Pappyandharriets.com | 192.124.249.118 | 971.236 | |
Wpsuperstars.net | 192.124.249.119 | 56.305 | |
Nyacknewsandviews.com | 192.124.249.119 | 1.822.885 | |
Abante.com.ph | 192.124.249.119 | 27.003 | |
Lenscratch.com | 192.124.249.12 | 332.734 | |
Platypusplatypus.com | 192.124.249.12 | 162.391 | |
Gamersnexus.net | 192.124.249.12 | 47.802 | |
Nub.ly | 192.124.249.12 | 617.579 | |
Givewp.com | 192.124.249.12 | 49.325 | |
Morinu.com | 192.124.249.12 | 1.233.970 | |
Wsko.org | 192.124.249.12 | - | |
Shaamtimes.net | 192.124.249.12 | 41.347 | |
Pakpa***ion.net | 192.124.249.12 | 206.097 | |
Lacsn.org | 192.124.249.12 | 3.991.864 | |
Onlinecasinoreports.com | 192.124.249.12 | 516.237 | |
Acenet.co.uk | 192.124.249.12 | 5.101.370 | |
Indiesunlimited.com | 192.124.249.12 | 215.718 | |
Linburytrust.org.uk | 192.124.249.12 | 5.122.867 | |
Bollywood.com | 192.124.249.12 | - | |
Oliveplanet.in | 192.124.249.120 | 406.202 | |
Bancounion.com.bo | 192.124.249.120 | 41.142 | |
Richfield.ac.za | 192.124.249.120 | 339.415 | |
Thewholesomedish.com | 192.124.249.13 | 77.919 | |
Riovagas.com.br | 192.124.249.13 | 45.603 | |
s***tingplatz.com | 192.124.249.13 | 1.440.687 | |
Businessideainsight.com | 192.124.249.13 | 130.193 | |
Hushhush.com | 192.124.249.13 | 154.997 | |
Reviewfantasy.com | 192.124.249.13 | 139.335 | |
Crazyforcrust.com | 192.124.249.13 | 80.930 | |
Softwarebattle.com | 192.124.249.13 | 64.822 | |
Writingitreal.com | 192.124.249.13 | 2.342.940 | |
Musicinaustralia.org.au | 192.124.249.13 | 7.201.036 | |
Sawpa.org | 192.124.249.13 | - | |
Fightforthesoulofthecities.com | 192.124.249.13 | - | |
Allaboutbison.com | 192.124.249.13 | 3.955.618 | |
Grokshare.com | 192.124.249.139 | - | |
Gertec.com.br | 192.124.249.14 | 1.030.241 | |
Bonoperuunido.com | 192.124.249.14 | 117.359 | |
Aeronet.cz | 192.124.249.14 | 48.411 | |
Courtauld.ac.uk | 192.124.249.14 | 95.190 | |
Cruisebusiness.com | 192.124.249.14 | 5.133.571 | |
Occatholic.com | 192.124.249.14 | 1.358.105 |