AI In Assessments – Automated Item Generation

In the assessment space, items are the basis for building intellectual property. Different industries utilize various processes and techniques to ensure the uniqueness and quality of these items. But how do these industries provide the uniqueness, consistency, and quality of the items developed? That’s where Artificial Intelligence (AI) will play a vital role in meticulously weaving elements such as item type, item language, item content (includes stimuli, stem, and distractors), item meta-data, item difficulty, and taxonomy levels together.

Let us understand the item design, item template, and the power of Artificial Intelligence (AI) in creating new items and refactoring the existing items.

Factors considered for Item design:

  • Interactivity: How the test taker will interact with the Item (e.g., choice-based selection vs. free text response)
  • Response Format: The type of response expected or captured (e.g., multiple choice single response, multiple choice multiple responses, fill-in-the-blank with free text, and a few more)
  • Scoring: How the Item is scored based on the response (e.g., auto scoring, semi-auto scoring, and manual scoring)

Every Item will be crafted with the listed factors in mind, ensuring it captures all parameters for presentation, evaluation, and analysis.

Item template design:

Like architectural template guide construction, item template is the foundation for consistent and high-quality item creation. While each item type will have its unique template outlining specific mandatory and optional parameters, all items will share a set of standard parameters necessary for their organization and analysis.

  • Item Stimuli
  • Item Stem
  • Item Distractors (in case of a predefined list of responses)
  • Response placeholder (in case of open-ended responses)
  • Answer Key (in the context of Objective Items)
  • Model answer (in the context of subjective items)
  • Complexity
  • Taxonomy classifications
  • Meta-data

Having captured all required parameters listed, the effective use of AI will significantly support authors in building new items and refactoring legacy items.

Leveraging AI for Efficient, Effective, and Rapid Item Generation:

By generating a set of items based on a predefined knowledge model, AI can significantly improve the speed of item generation and reduce the manual effort required from SMEs. SMEs can focus more on refining and fine-tuning the AI-generated items to ensure they precisely align with the desired learning objectives and assessment outcomes. The AI-generated item pools can serve as a springboard for collaboration and fine-tuning.

  • AI can generate a broader range of assessment items from the Knowledge bank, saving time and resources. (a content model of related information about a particular subject/topic).
  • AI can be used to create distractors for objective item types, preventing guessing factors and promoting a deeper understanding of the topic. It can generate model answers for the items used in the auto or manual marking process.
  • AI can analyze existing items and suggest modifications to meet newer objectives and outcomes, such as changing stem content, difficulty levels, and taxonomy levels and introducing newer distractors.
  • AI can suggest alternative item types based on the objective and outcome of an existing item (convert a multiple-choice item into a fill-in-the-blank format while ensuring the same knowledge or skill is assessed).
  • AI can create meta-data for each Item based on the outcome and analytical parameters defined.
  • AI can analyze items to identify potential biases based on language, content, or difficulty level. It can be used to group equivalent items and mark them as enemy items.

The future of assessment lies in the balanced collaboration between human expertise and AI capabilities. By adopting AI in item generation, the assessment space can unlock the potential to increase efficiency, cost-effectiveness, consistency, speed of development, content semantics, and an impactful item pool for all stakeholders involved.

Read these blogs to discover the latest insights on AI in Assessments:

Unleashing the Power of Artificial Intelligence in Online Assessment Tools

The recent advancement in the realms of Artificial Intelligence has opened up frontiers for improvements in the way technology can transform education. One of the key areas in education where AI in technology can positively improve effectiveness is online assessments. From automated item generation to personalized assessments, AI has the potential to revolutionize the way we teach, assess, and learn. We will delve into this blog post, exploring opportunities and applications AI can bring for online assessment tools.

AI in Automated Item Generation

Text-based generative AI models offer the capability to automate the creation of item or question content for assessments. Additionally, multimedia AI models can be utilized to generate accompanying multimedia assets, enhancing the overall richness of the assessment content.

AI in Adaptive and Dynamic Test Generation

AI enables the creation of adaptive and dynamic assessments that can adjust in real time based on a student’s performance. These assessments provide a personalized learning experience, catering to each student’s individual needs and abilities. AI-powered assessment tools can identify areas where students require additional support and adjust the difficulty level accordingly, ensuring a more engaging and effective learning process.

AI in Practical Assessments

AI-powered bots can engage in role-play scenarios, providing students with a realistic and interactive assessment experience. This technology allows for more authentic and immersive assessments, replicating real-world situations students may encounter in their future careers.

AI in Automated Marking

AI-powered marking tools can analyze subjective responses, including written and spoken responses, with a high degree of accuracy. NLP (Natural Language Processing) models aid in assessing written responses, while speech and linguistic criteria are used to evaluate spoken responses. This automation not only saves time for educators but also ensures consistent and unbiased marking.

AI in Analysis and Reporting

AI can perform statistical and psychometric analyses to provide detailed insights into student performance and assessment quality. It can generate data visualizations and interpret results, enabling educators to identify strengths, weaknesses, and areas for improvement. This data-driven approach helps educators make informed decisions about curriculum, instruction, and assessment strategies.

AI in Plagiarism and Malpractice Detection

Automated marking tools can identify instances of plagiarism in written responses. Remote proctoring solutions powered by AI monitor students during online assessments, analyzing video, audio, and keystroke data to detect suspicious behavior. Post-test forensics can further investigate potential cases of misconduct, ensuring the integrity and credibility of assessments.

AI for Automating Quality Assurance

AI can identify items that are biased, unfair, or ambiguous, ensuring the validity and reliability of assessments. AI-powered quality assurance tools can also provide feedback to item writers, helping them improve the quality of assessment items and reduce the likelihood of errors.

Conclusion

In conclusion, AI is revolutionizing the field of education and assessment. From adaptive testing to automated marking, AI-powered tools are transforming our approach to teaching and learning. AI’s ability to provide personalized, dynamic, and accurate assessments enhances the learning experience for students and allows educators to focus on providing high-quality instruction. As AI continues to advance, we can expect even more innovative and transformative applications of this technology in the realm of education and assessment.

Harnessing Generative AI for Efficient Test Data Generation

In the realm of software development and testing, the availability of high-quality test data is paramount. However, manually creating test data can be a time-consuming and laborious task, often leading to bottlenecks in the testing process. Generative AI, with its ability to produce realistic synthetic data, offers a solution to this challenge. In this article, we explore how generative AI revolutionises test data generation by automating the process, improving data quality, and accelerating the overall testing timeline.

Benefits of Using Generative AI for Test Data Generation:

1. Automation:

Generative AI automates the test data generation process, eliminating the need for manual data entry and reducing the associated time and effort. This automation enables developers and testers to focus on higher-value activities, such as improving the quality of test cases and analysing test results.

2. Improved Data Quality:

Generative AI algorithms can be trained on real-world data, allowing them to generate test data that closely resembles the actual input data. This leads to higher-quality test data that better reflects the scenarios encountered in production environments, improving the overall effectiveness of testing.

3. Increased Data Volume:

Generative AI can generate vast amounts of test data in a short time, addressing the challenge of data scarcity in testing. This enables thorough testing across multiple scenarios and edge cases, ensuring that the software application is robust and reliable under various conditions.

4. Improved Test Coverage:

Generative AI helps achieve broader test coverage by generating a diverse range of test data. This helps identify more defects and ensures that the testing process is thorough and comprehensive, reducing the likelihood of undetected issues in the software application.

5. Cost Reduction:

Automating the test data generation process and improving data quality leads to cost savings in the overall testing effort. By eliminating the need for manual data creation and reducing the time spent on testing, organisations can allocate resources more effectively and focus on innovation.

Generative AI has emerged as a powerful tool for test data generation, offering numerous benefits such as automation, improved data quality, increased data volume, enhanced test coverage, and cost reduction. By harnessing the capabilities of generative AI, organisations can streamline their testing processes, improve software quality, and accelerate the overall development timeline. As generative AI continues to evolve, it is poised to revolutionise testing methodologies and contribute significantly to the delivery of high-quality software applications.

Unveiling the Hidden Biases: A Deep Dive into Bias in Artificial Intelligence

As AI rapidly transforms various aspects of our lives, it is imperative to address the latent biases that can permeate AI systems. This article delves into the nuances of bias in AI, exploring its sources, implications, and potential solutions to mitigate its impact.

Sources of Bias in AI:

1. Data Bias:

The crux of bias in AI often lies in the data used to train AI models. If the data is biased, either intentionally or unintentionally, it can lead to biased outcomes. For example, training data that lacks diversity in terms of gender, race, or socioeconomic status can perpetuate existing biases in AI systems.

2. Algorithmic Bias:

AI algorithms, designed by humans, can inherently harbour biases. Hard-coded assumptions or preferences can lead to biased decision-making. For instance, an algorithm designed to predict recidivism rates may be biased against certain demographics due to historical data that overrepresents particular groups.

3. Human Bias:

As AI systems are developed and deployed by humans, they can inadvertently inherit biases from their creators. Conscious or unconscious prejudices, cultural norms, or personal experiences can influence the design, implementation, and evaluation of AI systems, leading to bias.

Implications of Bias in AI:

1. Unfairness and Discrimination:

Bias in AI can result in unfair and discriminatory outcomes, particularly for marginalised groups. For example, biased AI systems in hiring algorithms may lead to discrimination against certain demographic groups, limiting opportunities for employment.

2. Lack of Trust and Accountability:

When AI systems exhibit bias, trust in their outcomes can erode. This lack of trust can hinder the adoption and utilisation of AI technologies, diminishing their potential benefits. Additionally, it can be challenging to establish accountability for biased outcomes, as the complexity of AI systems often makes it difficult to trace the origin of bias.

3. Ethical and Societal Concerns:

Bias in AI raises profound ethical and societal concerns. Unchecked biases can perpetuate systemic inequalities, exacerbate social divisions, and undermine fundamental human rights. The implications extend beyond individual cases of bias, potentially affecting entire communities and shaping societal structures.

Mitigating Bias in AI:

1. Data Curation and Preprocessing:

Addressing bias in AI begins with the data used to train models. Techniques such as data augmentation, resampling, and algorithmic fairness can help reduce bias in training data, leading to more equitable outcomes.

2. Algorithmic Auditing and Fairness Evaluation:

Regularly auditing AI algorithms for bias is crucial. Fairness metrics and evaluation frameworks can help identify and quantify bias in AI systems, enabling developers to take corrective measures.

3. Human Oversight and Regulation:

Human oversight and regulation play a vital role in mitigating bias in AI. Establishing clear guidelines, standards, and accountability mechanisms can help ensure that AI systems are designed and deployed responsibly, minimising the potential for bias.

4. Education and Awareness:

Promoting education and awareness about bias in AI is essential. AI professionals, policymakers, and the general public must understand the sources and implications of bias to drive collective action towards more ethical and inclusive AI development.

Conclusion:

Bias in AI is a multifaceted issue that requires a multidisciplinary approach to address. By recognising the sources and implications of bias, stakeholders can work together to develop responsible AI practises, ensuring that AI technologies benefit all members of society fairly and equitably. The journey towards unbiased AI is an ongoing process that demands continuous vigilance, collaboration, and commitment to ethical AI development.

AI in Education Technology

At Excelsoft, we have recognised the need to take up an organisation-wide initiative to provide thrust and impetus to artificial intelligence and other related technologies. We have not only been emphasising the need for an ‘urgent AI push’ but have also been working in a clear direction in this regard.

There have been a variety of activities and initiatives that are already underway across the organisation. AI-driven features and functionality have been introduced into our products. There have been several proofs-of-concept (POC) of using AI tools and technologies. Our sales team has been positioning these products and solutions for our customers. We are conducting workshops and seminars frequently to keep all teams up-to-date on the latest developments in AI. Many departments have incorporated AI tools into their functions. Many of our colleagues have been using AI in their day-to-day work.

While all these actions are encouraging, we realised that there is a strong need to formulate a clear plan for an ‘organisation-wide AI strategy’ and execute the same in a structured manner as a programme. Earlier this year, we created an AI Focus Group at Excelsoft that will take ownership of this programme.


AI Focus Group at Excelsoft

The AI Focus Group focuses on the following core objectives:

  • Organisation Level Guidelines: For use of AI and ML in Products and Services
  • Alignment of AI Initiatives: Across all Products and Services
  • Capability Building: For Product, Tech and Operations Teams
  • Standard Tools and Libraries: Validated set of Tools and Libraries for Product, Tech and Operations Teams
  • Prototypes and Features: For Customer and Market-Driven Use-Cases
  • Legal and Ethical Considerations: For use of AI in Products and Services

AI in Processes and Operations

We have started incorporating AI into our processes and operations. Below are some key areas where we have been successful:

  • Development: AI-Assisted Code Generation, Review, and Correction has clearly increased the productivity of our developers across technologies. We are working closely with Microsoft and Amazon, exploring their tools in this effort.
  • Quality Assurance: AI-Assisted Generation of Test Scripts, Test Plans, and Test Cases has increased the overall quality and efficiency of our QA Teams.
  • Deployment: AI-Assisted Automated Deployments and Issue Prediction are being experimented with for seamless, issue-free deployments.
  • Technical Documentation: AI-Assisted Specifications Documentation and user manuals are being tried out for quick, accurate technical documentation.
  • HR and Learning and Development: The HR teams are actively using AI tools for screening CVs and AI-Assisted Policy Documentation.

AI Features in Products

Over the past few months, we have worked with our customers and gathered insights from forums and industry experts to identify key market needs that can be solved by applying AI. Below are some active projects we have been undertaking:

Online Assessments:

  • AI-Assisted Item and Test Construction
  • AI-Assisted Auto-Marking of Written, Spoken and Programming Responses
  • Smart Post-test Forensics
  • Smart Flagging of Duplicate and Compromised Items

Remote Proctoring:

  • AI in Remote Proctoring: Face, Voice, Object, Keystrokes Detection
  • Selective blurring of test content for proctors

Learning Experience and Content Services:

  • AI-Assisted Content Generation and Beautification
  • AI chatbots for Content Discovery and Tutoring
  • AI for Student Profiling and Personalised Learning
  • AI-Powered Content Recommendations

Student Success and Higher Education:

  • Predictive Analytics and Early Alerts
  • Detection of At-risk Students

Technologies and Models

Excelsoft has been an early adopter of the OpenAI platform and APIs. We have extensively used ChatGPT 3.5, GPT 4.0, and DALL 3.0 in our prototypes and features. In addition to OpenAI, we have explored other well-known LLMs:

  • Llama 2 (from Meta)
  • Bard (from Google)
  • Stable Diffusion (from Stability.ai)
  • Titan (from Amazon)

Risks and Challenges

While AI has in general shown great promise in various applications, we are also fully aware of the potential risks and concerns associated with its use, including bias, fairness, misinformation, fake content, ethical concerns, security threats, and legal and privacy issues.

We continue to observe this space closely. We participate in several AI communities and are working on setting up strong guidance principles for the use of AI in our products and services. We have also taken several measures to prevent any violation of the legal or ethical considerations of our employees and customers.

Moving forward, we are committed to executing our organisation-wide AI strategy in a structured manner.

The Age of AI in Education: Promises and Concerns

Remember AlphaZero that defeated Stockfish, which was the best chess program in the world? In 2017, it won 28 games, drew 72, and lost none. The next year, while playing 1000 games against Stockfish, it won 155, lost 6, and the rest were held a draw.

So what was so special about AlphaZero?

AlphaZero had no predefined moves or strategies from human play. It was a genuine product of Artificial Intelligence (AI) training where developers keyed in the chess rules and coded the program to develop strategies that improve the chances of winning every match. Just a self-training of 4 hours made AlphaZero the world’s most effective chess module. No human has ever beaten it.

It’s time we take AI and Machine Learning (ML) seriously. They are showing immense potential to create new types of businesses across business operations with striking results.

To check the popularity, I immediately started my research with Google. Searching “What is Human intelligence?” fetched 10 billion results in 0.42 seconds, whereas “What is Artificial Intelligence?” generated about 5.8 billion results in just 0.52 seconds.

The day is not far when AI (Inorganic Intelligence) will leave Human intelligence(Organic Intelligence) behind.

Welcome to the new world, where organic is meeting the inorganic.

AI is making that happen everywhere, and it’s viral!

AI is not an industry or a domain, let alone a single product. It is an enabler that has the capacity to learn, evolve, and surprise. It will disrupt and transform the human experience to levels that were not experienced up until now.

So, can we call this a new age?

As you know, human civilizations progressed with Stone Age, Bronze Age, and Iron Age, developing competencies using the materials and technology of each of the ages. Till the end of the 20th century, humans with better cognitive abilities have enjoyed success at work. However, with computing advancements in the 21st century, we are experiencing a disruption that has embarked us on a new age of freedom, where information and decision come for free.

We live in the AGE of AI

Every day, everywhere, AI is gaining popularity, and one of the biggest beneficiaries might be education. While AI has the potential to revolutionize the way we think about education, there are still many challenges and concerns that need to be addressed.

By the way, the presence of AI in Education is significant, with a valuation of $4 billion in 2022, and is projected to expand over 10% CAGR from 2023 to 2032. AI in Education Market Statistics, Trends & Growth Opportunity 2032 (gminsights.com).

In our recent Townhall, almost every conversation on Education Technology converged to AI and its impact. A question that gained everyone’s attention was, “In Education, who should benefit the most from AI?

Our CEO’s response was very candid, “AI should impact the student experience and improve performance outcomes.” He also added that the other stakeholders will also get their share of the pie by analyzing the insights from student engagement.

Let me summarize the takeaway.

In the EdTech domain, AI should serve the “King” aka THE LEARNER.
When they benefit, other stakeholders will get a share of the pie.

The big question revolves around the impact that AI will bring on education.

On my way to Mumbai, I had a conversation with a principal. He was both excited and skeptical about AI and was seeking answers to his questions.

  • What does AI-enabled education look like?
  • Will AI replace human intellect or critical thinking?
  • What do AI-enabled assistants mean to children?
  • Will AI-based assessments used for shaping human action be allowed?
  • Will cheating be rampant and cripple the learning model in education?
  • Should we teach children how to frame compelling questions and AI will do the rest?

I explained to him that the integration of AI in education could provide dynamic learning environments that are accessible, engaging, effective, and offer personalized learning experiences with intelligent tutoring systems.

But I still saw the worry in his eyes. He said, “My friend, it is important to strike an equilibrium between technology and human interaction, ensuring safe and secured learning environments.”

His words made me think about the opportunities that AI can offer to students and also its ramifications. AI can:

  • Personalize learning with tailored learning interventions, real-time feedback and opportunity for graded practice.
  • Enhance learning support and prioritize learning interventions along with improved assessment quality.
  • Present immersive experiences and enhance learner participation in regulated real-world situations.
  • Increase accessibility for learners.
  • Save costs by automating difficult tasks and facilitating customized instructions.
  • Revolutionize smart content creation by generating tailored learning materials.
  • Enhance the ease of performing administrative tasks and improve the efficiency of learning delivery.
  • Provide access to educational resources, particularly for students with limited access.
  • Analyze data patterns by using AI algorithms to detect early warning signs and alert educators for timely interventions.

However, we also need to be responsible enough to address the following:

  • Strengthen the bias and Inequality algorithms with continuous monitoring. This will help remove the learning and assessment bias.
  • Address privacy and security concerns as AI-powered learning systems gather a wide range of student information, including their behavior, learning progress, and personal data.
  • Address the tech dependencies and their impact on learners’ critical thinking and problem-solving skills. Even a continuous upgrade of modern technology to ride this AI wave is worrisome.

AI has immense potential to revolutionize the education sector. However, legislatures, policymakers, entrepreneurs, educators, and technology enthusiasts must work together to ensure that AI-driven learning platforms and tools are used ethically and responsibly.

We at Excelsoft Technologies have a dedicated R&D team that has undertaken a keen interest in formalizing and expanding the use of AI technology and has been working on the following pointers:

  • Define policies and guidelines to guide the use of AI for all the stakeholders.
  • Publish ethical implications of using AI in education, including data privacy, security, misuse, and algorithmic bias concerns.
  • Research the latest trends and developments in education technology, including current applications, SAAS business models, and Integration APIs for potential future use cases.
  • Present the out look of different stakeholders in evaluating the potential benefits and risks of AI in education.
  • Develop specific manifestations of how AI is used in our demonstration school (Excel Public School) and evaluate its effectiveness.
  • Formalize the training and development needs of the stakeholders and ensure the effective incorporation of AI in the education system.
  • Define the process of monitoring and evaluating the ethical usage of AI.

In conclusion, AI holds great promise for the future and will play a pivotal role in shaping our lives. The integration of AI in education holds tremendous potential for transforming learning environments. By leveraging AI’s capabilities, education can become more accessible, engaging, and effective.

We at Excelsoft focus on the following to achieve good outcomes in the field of AI:

  • Ease of deployment with integration APIs and process data analytics
  • Integrating skills and competencies with strong learning analytics and reporting
  • Performance management with adaptive testing and evaluation
  • Promote responsible actions with state-of-the-art proctoring and evidence-based practices

For more information on our pursuit, reach out to us at connect@excelsoftcorp.com