Simulation in CAT research
Two primary simulation means can be found often: 1) real-data or post hoc; 2) Monte Carlo simulation.Real-data or post hoc simulation is used when CAT is to be used to reduce the length of a test that has been administered conventionally. The "item bank" used in this case would be all the items in a conventional test. The objective of applying CAT procedures is to determine how much reduction in test length can be achieved by "re-administering" the items adaptively, without significant changes in the psychometric properties of the test scores. The data are the item responses of a group of examinees on the conventional test that is being analyzed.
Monte carlo simulation can be used to evaluate the potential performance of various approaches to CAT with various populations and to evaluate the potential performance of CAT using using real or hypothetical item banks. This approach to simulation typically is used prior to the implementation of a live CAT testing program to evaluate the performance of a calibrated item bank and to determine operational CAT parameters, such as appropriate values of CAT entry theta values, termination criteria, and item exposure.
Monte carlo simulation differs from real-data simulation in the following characteristics:
- "Examinees" (generally referred to as "simulees") are generated by the simulation process to have specified distributions of theta.
- Item parameters can either be generated to have specified values and distributions, or item parameter estimates from real items can be used as the CAT item bank.
- Item responses of the simulees are generated from an appropriate IRT model.
- CATs are then administered using prespecified CAT algorithm values to answer a specific research question or set of research questions.
posted by Shunkai Fu @ 5:42 AM
4 Comments:
Hi,thanks for your visiting. For the items, they need be calibrated first before they can be input into the bank. During experimental step, try more items since it will enable you find something you want. One basic rule, from my point of view even I never realize commercial item bank, is that the difficulty distribution should be wide enough so that your selection rule will have more choices.
Hi
Thanks for your reply .
So for an item to be added in to a bank,is it enough that the items should be calibrated ie should have difficulty,discrimination and guessing parameter values if IRT kind of calibration is used ...
Hi, you are right. For CAT based on IRT, you need calibrate the a, b, c respectively for each item. Then, these parameters can be used to select the next optimal item during adaptive testing procedure. Of course, for real CAT exam, b needs more consideration considering that distribution should be reasonable.
When items are in bank, you are on the normal way described in most papers: select item, make judgement, and stop the exam.
Post a Comment
<< Home