Reliability of CAT
How can we trust computerized adaptive testing? For a novel thing, there is indeed a long period to be trusted, and CAT is not different. Naturally, we will compare result of CAT with traditional paper based test, and the ideal outcome is that examinees be assigned same or similar scores if they participate these two kinds of tests. From classical true-score theory, we know that true score of each examinee is unkown. So, we need think of some reliable ways to evaluate our work on CAT.
posted by Shunkai Fu @ 1:59 PM
2 Comments:
In item response model, aberrant response pattern will result in no-existence of maximum likelihood: some relatively difficult items be answered correctly while easier ones incorrectly. I don't know if this problem has been solved. It is really a trivial problem since easy and difficult are always relative even we can declare that item's difficutly level can be determined through pilot testing of big population. We all know GRE, and lots of Chinese examinees complain the items presented in CAT, which shows that items easier for Enlgish-speaking people may, and very likely, will be difficult for non-English examinees.
I sincerely feel aberrant response patterns should not exist in educated test-takers. Aberrant response is indicative of the individuals lack of understanding of the test or his/her lack of interest in the test. But, if the same test is used for university-level testing for regular degree courses, aberrant response patterns are certainly of question.
Also, I haven't come across an alternate approach to Maximum Likelihood estimation. MLE is used to locate the turning points of the information curve, is it not? So, any alternate method to locate the roots of a equation should be able to replace the same. Now, why nothing much has been done on this front is of question!
Post a Comment
<< Home