OCV - Assessment Item Analysis
- Brooke
- Sturdy Knight
- George Dale
Topic Overview
Item Analysis provides statistics on overall test performance and on individual test questions. These data help faculty recognize questions that might not adequately discriminate between students who understand the material and those who do not. Faculty can use this information to improve questions for future tests or to adjust credit on current attempts. Ineffective or misleading questions are identified easily, corrected in the Test Canvas, and are re-graded automatically.
Steps
You can run item analyses on tests that include single or multiple attempts, question sets, random blocks, auto-graded question types, and questions that need manual grading. For tests with manually graded questions that have not yet been assigned scores, statistics are generated only for the scored questions. After you manually grade questions, run the item analysis again. Statistics for the manually graded questions are generated and the test summary statistics are updated.
Go to one of the following locations to access item analysis:
A test deployed in a content area
A deployed test listed on the Tests page
A Grade Center column for a test
Click the drop-down arrow next to the test name.
Select Item Analysis.
In the Select Test drop-down list, select a test. Only deployed tests are listed.
Click Run.
View the item analysis by clicking the new report's link under the Available Analysis heading or by clicking View Analysis in the status receipt at the top of the page. You will also receive an email saying that the analysis is available.
Completed test analysis view link
Test Summary Statistics
The summary statistics at the top of the Item Analysis Page provide data on the tests as a whole:
Possible Points - the total number of points for the test
Possible Questions - the total number or questions in the test
In Progress Attempts - the number of students currently taking the test that have not yet submitted it
Completed Attempts - the number of submitted tests
Average Score - scores denoted with an * indicate that some attempts are not graded and that the average score might change after all attempts are graded
Average Time - the average completion time for all submitted attempts
Discrimination - this area shows the number of questions that fall into Good (greater than 0.3), Fair (between 0.1 and 0.3), and Poor (less than 0.1) categories. A discrimination value is listed as Cannot Calculate when the question's difficulty score is 100% or when all students receive the same score on a question. Questions with discrimination values in the Good and Fair categories are better at differentiating between students with higher and lower levels of knowledge. Questions in the Poor category (less than 0.1) are recommended for review.
Difficulty - this area shows the number of questions that fall into the Easy (greater than 80%), Medium (between 30% and 80%) and Hard (less than 30%) categories. Difficulty is the percentage of students who answered the question correctly. Questions in the Easy or Hard categories are recommended for review.
Note: only graded attempts are used in item analysis calculations. If there are attempts in progress, Item Analysis ignores those attempts until they are submitted and you run the item analysis report again.
Question Statistics Table
You can filter the questions table by question type, discrimination category, and difficulty category.
In general, good questions have:
Medium (30%-80%) difficulty
Good or Fair (greater than 0.1) discrimination values
The table provides the following statistics for each question in the test:
Discrimination - indicates how well a question differentiates between students who know the subject matter and those who do not. Values can range from -1.0 to +1.0 and are calculated using the Pearson Correlation Coefficient. A discrimination value of less than 0.1 or that is negative indicates that the question may need review. Discrimination values cannot be calculated when the question's difficulty score is 100% or when all students receive the same score on a question.
Difficulty - the percentage of students who answered the question correctly. Difficulty values can range from 0% to 100%, with a high percentage indicating that the question was easy. Questions in the Easy (greater than 80%) or Hard (less than 30%) categories might need review.
Graded Attempts - number of question attempts where grading is complete. Higher numbers of graded attempt produce more reliable calculated statistics.
Average Score - if a question has attempts that need grading, only the graded attempts are used to calculate the average score.
Standard Deviation - measure of how far the scores deviate from the average score. If the scores are tightly grouped, with most of the values being close to the average, the standard deviation is small. If the data set is widely dispersed, with values far from the average, the standard deviation is larger.
Standard Error - an estimate of the amount of variability in a student's score due to chance. The smaller the standard error of measurement, the more accurate the measurement provided by the test question.
How to View Question Details on a Single Question
You can investigate a specific test question by accessing its Question Details page. This page displays student performance on the individual test question you selected.
On the Item Analysis page, scroll down to the question statistics table
Click on a question title to display the Question Details page