| Yendor 2003-02-21, 6:24 pm |
| Your question requires some background to answer fully. The process has
changed as the MCSE program has evolved. Here's the exam writing process in
a nutshell:
1. Define the domain
This is done internally at Microsoft by people from several different
groups, (with input from external experts.) Often the Official Curriculum
folks are involved, also the people from the Cert team, of course, and
people from the development teams. They go through and essentially try to
anticipate (remember that exams are created for products that are not yet in
production) what tasks an expert would need to be able to do. This becomes a
huge outline and the basis for step 2. Also, they try to weigh how important
each item is. If something is done often but is not very challenging (adding
a new user for example) it might get fewer questions than something that is
not done as much but is more difficult (adding new servers.)
This domain list is reviewed by people internal to MS and external. When the
domain is defined, they move on to step two.
2. Write the questions
This is often outsourced to a company that specialize in test writing. They
go through and write questions based on the domain and apply Bloom's
taxonomy to determine at what "level" to write the questions. (Bloom's
taxonomy is beyond the scope of this answer, but if your interested,
typically they shoot for questions in the Comprehension and Analysis range.
http://www.coun.uvic.ca/learn/progr...outs/bloom.html )
3. Review the questions
The questions are reviewed by internal experts. Some of the same people that
helped created the domain. It can often take 20 minutes to review a single
question. And it will either be A) Accepted, B) Sent back to the test writer
to fix, or C) Discarded. (Note: Microsoft used to use the discarded
questions as their practice exams. A practice that mercifully has stopped.)
This step takes several iterations. It's not unusual to review a question 2
or 3 times. Although if it's still not fixed after 3 tries, it's often
unfixable.
(Oh, I almost forgot the psychometrician. This is a guy who often knows
nothing about the subject of the test, but evaluates the test for common
test errors. For example, if the correct answer is twice as long as the
incorrect answers, the psychometrician will catch that.)
4. Hold the Beta
Most of the experienced people on this site probably already know that when
you sign up to take a Beta exam, there is no "passing" score. That's because
Microsoft uses the results of the beta to determine the "cut score." The
beta will often have all the questions from the test bank.
5. Hold the cut score meeting
The cut score is the pass/fail score. I've seen the cut score expressed both
as a percentage (81%) and as a raw number (651/800.) The results of the beta
are analyzed and decisions are made on an individual question basis. For
example if 75% of the people missed a question that had 4 possible answers.
AND if the 25% that got it correct did NOT score high on other sections of
the test, the question is thrown out. However, if 75% of the people missed a
4 parter and the 25% that got it right also scored high elsewhere on the
test, the question is kept. (In the first case, the 25% was because people
guessed. In the second it was because they knew the material.) In a similar
vein if 99% of the people get a particular question correct, it's thrown
out. The whole point of the test is to determine who knows the material and
who doesn't. A question that everyone misses or that everyone gets correct
isn't valuable as a determiner.
After the decision has been made on which questions to keep, the remain
results are re-analyzed to determine where the final cut score should be.
There is no exact science to this part. I've never heard anyone say, "X% of
the people should pass the beta so we'll put the cut score here." Instead,
the data tends to show some natural separation between people who obviously
knew the answers and people who didn't.
So, to sum up, each exam has it's own maximum score based on the number of
questions. (Although if it's normalized to a percentage it's 100%,
obviously.) How the exams are scored involves a lot of different people, a
healthy dose of statistics and lots of pizza and coke at the marathon cut
score meetings.
Hope this we helpful.
Rodney Bliss
Purple Crayon Consulting
"Nick Funk" <nfunk@rtconline.com> wrote in message
news:3E46C6B5.3000109@rtconline.com...
> I have seen in many different locations the minimum passing score for
> each of Microsoft's exams.
> But I have never seen the maximum score or how the exams are scored.
>
> Does anybody know the answers to this question?
>
> Thanks,
> Nick
>
|