Adoption of Linear Optimization for Automated Test Construction
Introduction
Constructing a test involves much more than simply selecting questions to fill a set amount of time. It requires meeting numerous constraints to ensure the assessment accurately measures what it is intended to. Traditionally, test authors have to manually juggle these constraints to create a balanced set of questions. However, this process often leads to suboptimal results, where the selected items may not fully align with the test’s objectives. Additionally, managing the complexity of multiple constraints can be time-consuming. Enter linear programming—a powerful mathematical approach frequently employed in industries for optimization and decision-making. By leveraging linear optimization techniques, we can revolutionize the way tests are constructed, creating automated, efficient, and goal-oriented assessments with greater precision.
Objectives and Constraints
Binary Linear Programming helps in selecting the best test questions by using user-defined objectives and constraints. Here’s what they mean:
- Objective: A goal that requires an attribute to reach a minimum or maximum value from the available pool of questions.
- Constraint: A requirement that an attribute (like test duration, information, or word count) stays within specific limits. For example, the test should have between 58 and 62 questions, and the word count should fit within certain bounds.
Example of Constraints at Various Levels
Test Level Constraints:
- Number of lines/words in the test.
- Total number of items not exceeding the prescribed limit.
- Total time (sum of average times for items) not exceeding the allocated exam time.
Item Level Constraints:
- Excluding enemy items (items that can’t appear together).
- Ensuring item parameters are within a specific range.
- Limiting the number of question types in the test or in each subsection (like History or Geography).
- Including logically related items (if one item is selected, another related item must also be selected).
Item-Set Constraints:
- Ensuring a certain number of items from each subsection (like History or Geography).
- Other constraints might include the type of questions, the author, or the presence of additional materials like graphs or tables.
Example of Objectives and Constraints
The Test Information Function (TIF) in Item Response Theory (IRT) is important in test construction. TIF shows how well the test measures a student’s ability at a certain level. The goal is to minimize variations from the desired TIF. Objective: Minimize y (the allowable tolerance).
Constraint List:
- Test Information Tolerances:
a. (i11*x1 + i21*x2 + i31*x3 + i41*x4 + ——- + in1*xn ) <= I1 + y _________ (1) b. (i11*x1 + i21*x2 + i31*x3 + i41*x4 + ------- + in1*xn ) >= I1 – y _________ (2)
‘y’, is the allowable tolerance.
Two constraints for each point based on tolerance, are set on a particular theta value (say at -3).
Where,
i11 is the information of item1 at the 1st set point.
i21 is the information of item2 at the 1st set point.
in1 is the information of item n at the 1st set point.
and,
x1, x2, x3….. xn indicates the binary values of whether the items in the pool are to be considered for the test or not. N is the total number of items to be picked for the test. - Number of items in the test(N)x1 + x2 + x3 + —- + xn = N ___________(3)For the sake of simplicity, only 3 equations are presented. In a real ATA scenario, there will be multiple linear equations that need to be solved to arrive at a solution that satisfies all the constraints that were set. Finally, a set of binary values x1, x2, x3… xn are obtained which indicates the items from the pool to be picked for the test.
Conclusion:
Implementing linear optimization for automated test construction significantly simplifies the process while enhancing accuracy. By utilizing mathematical models, this approach selects the most suitable questions from a vast item pool, ensuring that all test constraints—such as the number of questions and time limits—are met with precision. This not only saves educators valuable time and effort but also results in tests that are more balanced, fair, and aligned with their intended measurement goals. Ultimately, this innovative method enhances the reliability of assessments, creating a better experience for both teachers and students. It holds great promise for improving the overall quality of educational testing, benefitting all stakeholders.