Computerized Adaptive Testing

Can I Measurement Software?

Use IRT or related measurement theory, we can measure the latent traits of human beings. Being an object in the nature, like all other objects, software has some similar metrics, like usability, robustness, and so on, and if we could measure these index by using theory used in adaptive testing. Through given a series of items adaptively, our system can determine the hidden level of one examinee; similarly, can we do the same thing on software? At least, we can determine if two software of the same type, like IE and Netscape, have the same strength in specific direction. Steps involved can be listed as follows (based on measurement decision theory):
1) Design item bank;
2) Pilot testing on a series of software, and get the probability of Master and Non-master in the population of a group of software given cutting score; Also, calculate the probability to satisfy or non-satisfy specific item given Master and Non-master state;
3) Use the calibrated items on potential examinee - softwares that we are interested if they are Master or Non-master;
4) Selection rules and stopping rules are the same;
5) After a length of items, we know the softwares' state: Master or Non-master.

Of course, we can add more levels of state, like Excellent, Good, Mature, Just-Pass, Need Enhancement, Forgetable, Fail.

posted by Shunkai Fu @ 6:05 AM