Wednesday, May 01, 2013

Thoughts on the ISTQB Open Letter

Since it is hard to elaborate a response in 140 characters, I am blogging this. First, full disclosure, I am on the board of directors of the ASTQB and I am a training provider of ISTQB certification courses, as well as over 60 other courses I have written. (I make a small percentage of my income from ISTQB training.) I was also a founding author of the CSTE test certification from QAI. This is from my own perspective and does not represent the views of the ISTQB or ASTQB.

Second, I really believe as testers we have the right and responsibility to ask questions. So my issue is not with the asking of questions. We can still be friends.

I tweeted yesterday that "The open letter to the ISTQB is meaningless." Here's why.

1. The letter fails to distinguish between the ISTQB and the country examination boards (such as the ASTQB, Canadian Testing Board, UKTB, etc.) that write and administer the exams. Therefore, the ISTQB does not write exam questions. To get the answers to the questions about the validity of questions and coefficients, you would have to write such a request to each individual country exam board. The ISTQB may provide a high-level response, but it can't answer the detailed questions posed because it simply doesn't have the information.

2. There are Non-disclosure Agreements in place to protect the intellectual property of each country board and the ISTQB in general. One of the challenges with any exam (ITIL, PMP, etc.) is to keep the contents of the exam confidential as to prevent questions from being passed around. There is a sample exam available on the ASTQB website that gives a flavor of the questions being asked. Kryterion would not release results because they also have confidentiality restrictions.

3. "Have there ever been any problems with the validity of the exams?" This is like asking, "Are you still beating your wife?" Everything has shortcomings. The reason the ASTQB invested in independent exam reviews and measurement was to make the exam as valid as possible.

I agree that the questions on the exam must be a valid reflection of the learning objectives in the syllabus.  As a training provider, I don't know what is on the exam. There is a firm line of separation. I focus on the methods and application as opposed to the brain cram approach.

That's it for now. Gotta get back to the test lab.

Thanks,

Randy





7 comments:

Dot Graham said...

Thanks for your comments on this Randy - good points!

TestSheep said...

Glad to see your posts - I think everyone welcomes to hear another side to the discussion.

Firstoff, I'm not necessarily anti-certification. But I think like many I have concerns about it.

The problem I have though with what you've said about "an open letter to the ISTBQ is meaningless" is that the ISTQB states to be an organisation in the interests of testers. However if we go through the points above, it means a tester such as Keith Klain or even myself cannot in any way engage with the ISTQB itself?

By the way - hilariously - your comment moderation caused a little offence to my Scottish heritage ...

https://twitter.com/TestSheepNZ/status/330436020329914368/photo/1

I think maybe you need to watch Braveheart as penance.

Randall Rice said...

Thanks for your comments!

What I'm trying to point out above is that there are reasons that the data requested is unobtainable from the ISTQB. The larger question/concern of exam validity is certainly a valid one.

The ISTQB is an umbrella organization that relies on volunteers in over 40 country boards. The governance committee, or executive committee of the ISTQB can provide a response, but since there is no central exam, the responsibility for validity rests with each country board that produces an exam. (Some boards license exams from other boards.) BTW, that's why it's dangerous to study for the U.S. exam (or N.Z. exam, etc.) by finding sample exams on the web.

I agree that testers should have a way to engage with the boards. I try to be available to listen to testers' concerns and trust me, I pass them along at board meetings. :-)

No offense intended on comment moderation. I have to control the spam. I also have a Scottish heritage, so watching Braveheart is always inspiring.

The last two days here in Oklahoma were cold, windy and rainy. The exact conditions we experienced the day we toured Stirling Castle! But is was fun anyway.

Feel free to write me any time. Thanks for taking the time to comment!

Randy.

TestSheep said...

Thanks for the response - some interesting points. I haven't signed that petition based on the fact that they're not really questions I'd really consider such "hot topics". However I would like to think that ISTQB on whatever level would be able to have questions asked of it.

To me the very essence of testing is about being able to weather the many questions that are asked of us - I'd expect a board who represented such a profession to be able to do the same.

No deep offence taken at the comment moderation, but I did think "Captcha - did you test this?".

Just saw your "Testing Dirty Systems" - you need to work on a Kindle format for those of us in far flung countries for whom postage is CRIPPLING.

Randall Rice said...

Sure, I think questions are good as long as they are answerable and have value. I think Cem Kaner's response made some good points about what is reasonable for a company or organization to disclose about its own QC data. http://context-driven-testing.com/?p=80

On the Captcha, I'm a Google's mercy, I think. I have looked into changing the text, but no answer yet.

On the Testing Dirty Systems book, I am close to finishing a revision and conversion into Kindle and other digital formats. I buy most of my books these days either on Kindle, or read them on my Safari subscription.

All the best,

Randy

Richard Robinson said...

Thanks Randy for providing a channel to discuss the current *hot topic* going around twitter etc.

I read your post with interest, and one thing struck me right away. In your first point, you say that the 'open letter' does not specify which chapter of ISTQB is in question. That's fair enough, but isn't the glossary and syllabus the same internationally?

So doesn't this open up two problematic possibilities:
1) The exam is set locally but the correctness of each answer is still driven by the glossary and syllabus? We know this to be true because you teach the syllabus/glossary in preparation for the exam. If this were not true, then ethically you would probably have to refuse to offer the teachings you do.

2) The exam is based on the questions which come from local testing practitioners, reflecting the local testing context? We know this to be true also, because we see that ISTQB ask for exam questions from tester.

Does that mean the exams are dependent on local industry context but they are graded against international rules? This sounds like it could get rather tricky to obtain questions that fit the local chapter, but also do not contradict any other local chapter. Therefore, it would seem impossible for the ISTQB to NOT be involved, don't you think? The global outfit would be forced to maintain all questions I would reasonably conclude.

I don't quite get it yet. You can help me. It seems too easy to just say that the exams are set locally, and therefore the ISTQB is not involved. There is definitely a clear syllabus link and local context input there don't you think?

Richard Robinson

Randall Rice said...

Hi Richard,

Thanks for your comments and for the way you expressed them. Those are great questions and I'll be happy to tackle them. Let me also say that it took me awhile to understand how this works!

It is very important to know that the ISTQB does not write exam questions. That is left to national exam boards. The way it works in the ASTQB is that the questions are written by a Technical Advisory Group (TAG). The questions must be based on a learning objective. All the LOs are elaborated in the syllabus.

The syllabus is common worldwide. The ISTQB DOES create all the syllabi through working groups comprised of people from national boards. Therefore, when a course is accredited by a national board, or an exam created by a national exam board, the criteria is the syllabus (Foundation, Advanced, etc.).

You are correct in that if a particular exam deviated from the syllabus content and learning objectives, that would be a problem to training providers and students. In the ASTQB, we measure exam performance.

There are ISTQB-written exam guidelines to help national exam writers achieve some consistency. The cultural differences are limited to things like currencies, weights and measures, etc. Those national differences are not related to applying testing methods in ways not described in the syllabus. BTW, the syllabus is not intended to be prescriptive. So, a training provider may very well teach a certain technique differently than another TP might. Both need to convey the same concept as expressed in the syllabus.

Personally, I focus on teaching the concepts that form the syllabus instead of having people memorize lists. (Although, there are some places that is needed, such as the K1 objectives.) The higher level LOs, such as K2 - K4 on the Foundation exam REQUIRE understanding of the rational behind the techniques. The K3 LOs REQUIRE that people are able to perform the technique correctly to arrive at the correct answer.

For your second question, the answers and grading of exams are also performed by national boards. Some boards opt to license exams and answer sets from other boards. So, a question consists of both the question and the correct answer. All I can speak to is the ASTQB process, but the questions and answers are reviewed very carefully.

The ISTQB does not review any exams that I am aware of. That is all left to the national exam boards.

This is why I wrote my initial response - not to be snarkey about it, but to simply say the data as requested in the open letter isn't there. I do agree with Dr. Kaner that even if it did exist, it would be appropriate for it to remain private for the purpose of exam improvement.

Another way to say it, is that if an organization knew it's internal findings would be used against it, they might rightfully never conduct the study. Of course, regulated organizations and government entities may be required to release audit results.

I hope I didn't ramble on too much! I am always happy to clarify anything that I can.

Thanks,

Randy