Test Item Analysis (Activity)

    Background Information: A challenge when creating reliable and valid assessments is that the content of the questions need to not only be objective (based on the facts and current research of a topic/issue), they need to be effective too. In other words, the purpose of each question needs to clearly challenge the test taker's understanding of the content and the choices are not too difficult or too easy. It's the assessment equivalent of the three little bears: what's just right? Two methods to test the quality of your questions are the difficulty and discrimination indexes. The data that these indexes provide can help you identify what questions are showing validity and reliability, as well as identifying some questions that should be replaced. The difficulty index is used to compare how many students were able to answer the question. The formula has you divide the number of students who selected the correct answer by the total number of students taking the exam. The formula looks like this: Difficulty = Students with Correct Answer for the Question / Total Students For example, Table 1 illustrates how many students selected each answer choice for two questions on a multiple choice exam (answers were labeled A, B, C, D). Table 1 Question A B C D #1 0 3 24* 3 #2 12* 13 3 2 * Denotes correct answer. For Question #1, you may already at a glance notice that choice A was not a very good distractor (an answer that sounds likely but is not fully accurate; they are used to challenge the student to think critically)-- no one selected that answer. When we do the math and compute the difficulty, we're going to get more information. For Question #1, we divide the number of students who choose the correct answer (24) by the number of total students (30), which has a result of 0.8. Once calculated, the index score is going to indicate a few things that will impact how you evaluate the question: ○ Scores 0.75 or above: the question is very easy. It's possibly too easy since everyone knew the answer or was able to guess correctly. ○ Scores 0.25 to 0.75: the question is in the middle of the road, not too easy and not too difficult. It's challenging, but it's not so easy that you could guess and get it right. ○ Scores 0.25 or lower: the question is too hard. It's so challenging that students who scored relatively well struggled with identifying the purpose of the question or there was possibly more than one right/best answer. For Practice 1: Try calculating the difficulty index for Question 2 in Table 1, and see if you can calculate that question's index score (the answer is at the bottom of the page). The discrimination index looks at comparing how well the students who scored high on the overall test scored versus the students who had an overall low score. To get started, it's best to organize a chart listing each student based on their overall scores from highest to lowest, as well as additional columns for each question on the exam. For each question, you want to note whether a student then got the question incorrect (0) or correct (1). Next, split the class into an upper group (the top third of performers) and a lower group (the bottom third). Once you have determined how many students are in each group. Count the the number of students in the lower group who got the item correct and the number of students in the upper group who got the item correct. Subtract the total of the lower group from the total of the upper group, and note the difference between the lower and the upper group. Finally, you will divide the difference between the lower group from the upper group by half the total number of students in the class. The formula look like this: Discrimination = (Number of students in the upper group - Number of students in the lower group) / (Total number of students in the class / 2)* *complete the computations that appear in parenthesis first, then divide the totals. This is where most students run into an issue. When interpreting the discrimination index results, remember: ● Positive discrimination (0 to 1) means the upper group did better on the exam than the lower group; this is normal since it means the students who did well overall on the exam did well on this question; it means this question is in line with the rest of the exam. ● Negative discrimination (-1 to 0) means the lower group students did better on the exam than the upper group. This means that the overall most knowledgeable students are getting the item wrong and students who struggled more with the overall exam are getting the question right more often. It may indicate that you're measuring something other than what the rest of the test is measuring with this question. Lets take a look at Table 2 for a sample test item analysis. Table 2 Student Total Score (%) Questions 1 2 3 4 5 1 90 1 0 1 1 1 2 90 1 0 1 1 0 3 80 0 0 1 0 1 4 80 1 0 1 1 0 5 70 1 0 1 0 0 6 60 1 0 0 0 1 7 60 1 0 1 0 1 8 50 1 1 0 1 0 9 50 1 1 0 0 1 10 40 0 1 0 0 0 Table 2 features the overall scores for the students in the upper and lower group in a class of ten (10) students. In Question #1, there were four (4) students in the upper group who got it correct and four (4) students in the lower group, and the difference between them is zero (0). Since there are ten (10) students total, the number in each group is five (5). So, I would be dividing zero (0) by (5) to get my final result. After finishing this calculation, the discrimination index score is zero (0), which indicates positive discrimination, so this question is valid and reliable since it's on-topic. For Practice 2: Try calculating the discrimination index for Question 2 in Table 2, and see if you can calculate that index score (the answer is at the bottom of the page). Part 1: You're being asked to complete a test item analysis using Table 2. Follow these steps to determine the Difficulty Index and the Discrimination Index. 1. After the students are arranged with the highest overall scores at the top, count the number of students in the upper and lower group who got each item correct. 2. Determine the Difficulty Index by dividing the number who got it correct by the total number of students. 3. Determine the Discrimination Index by subtracting the number of students in the lower group who got the item correct from the number of students in the upper group who got the item correct. Then, divide by the number of students in each group (in this case, there are five in each group). 4. Record your responses for Questions 3-5 by completing the provided Table below. Item # Correct (Upper group) # Correct (Lower group) Difficulty (p) Discrimination (D) Question 3 Question 4 Question 5 After completing the computations, please answer these four questions based on your test item analysis: 1. Which question was the easiest? Why? 2. Which question was the most difficult? Why? 3. Which item has the poorest discrimination? Why? 4. Which questions would you eliminate first (if any) – why? Part 2: Write a two-page (2) reflection about how this exercise is or is not relevant to your practice. Consider the purpose of collecting and organizing data, or if there are alternate methods that would be equally or more relevant to your practice based on research. Reflection Length Requirement: Two (2) pages not including the completed chart Format Requirement: APA Research Requirement: two (2) secondary outside resources should be cited in-text and in the Reference Page (the articles provided in this unit do not count towards this requirement. You are encouraged to cite them in your paper, yet they will not count towards the secondary resource requirement).