How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures
The widespread adoption of Artificial Intelligence (AI) in K-12 education highlights the need for psychometrically-tested measures of teachers' AI literacy. Existing work has primarily relied on either self-report (SR) or objective-based (OB) assessments, with few studies aligning the two within a shared framework to compare perceived versus demonstrated competencies or examine how prior AI literacy experience shapes this relationship. This gap limits the scalability of learning analytics and the development of learner profile-driven instructional design. In this study, we developed and evaluated SR and OB measures of teacher AI literacy within the established framework of Concept, Use, Evaluate, and Ethics. Confirmatory factor analyses support construct validity with good reliability and acceptable fit. Results reveal a low correlation between SR and OB factors. Latent profile analysis identified six distinct profiles, including overestimation (SR > OB), underestimation (SR < OB), alignment (SR close to OB), and a unique low-SR/low-OB profile among teachers without AI literacy experience. Theoretically, this work extends existing AI literacy frameworks by validating SR and OB measures on shared dimensions. Practically, the instruments function as diagnostic tools for professional development, supporting AI-informed decisions (e.g., growth monitoring, needs profiling) and enabling scalable learning analytics interventions tailored to teacher subgroups.
Self-reported measures (surveys) are often not correlated or even negatively correlated w/more objective measures (such as observations, scenario/performance assessments). Examples:
* Teacher AI literacy
https://arxiv.org/abs/2601.06101* Applying professional development to the classroom
https://academic.oup.com/bioscience/article-abstract/61/7/550/266257* AI cognitive offloading
https://www.goedel.io/p/the-machine-that-stops-you-from-thinking* Student learning from teaching
https://www.pnas.org/doi/10.1073/pnas.1821936116* And grades
https://link.springer.com/article/10.1007/s10648-023-09819-0* TPACK
https://osf.io/preprints/psyarxiv/bhqxp_v2#EdDev #AIEdHow to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures
The widespread adoption of Artificial Intelligence (AI) in K-12 education highlights the need for psychometrically-tested measures of teachers' AI literacy. Existing work has primarily relied on either self-report (SR) or objective-based (OB) assessments, with few studies aligning the two within a shared framework to compare perceived versus demonstrated competencies or examine how prior AI literacy experience shapes this relationship. This gap limits the scalability of learning analytics and the development of learner profile-driven instructional design. In this study, we developed and evaluated SR and OB measures of teacher AI literacy within the established framework of Concept, Use, Evaluate, and Ethics. Confirmatory factor analyses support construct validity with good reliability and acceptable fit. Results reveal a low correlation between SR and OB factors. Latent profile analysis identified six distinct profiles, including overestimation (SR > OB), underestimation (SR < OB), alignment (SR close to OB), and a unique low-SR/low-OB profile among teachers without AI literacy experience. Theoretically, this work extends existing AI literacy frameworks by validating SR and OB measures on shared dimensions. Practically, the instruments function as diagnostic tools for professional development, supporting AI-informed decisions (e.g., growth monitoring, needs profiling) and enabling scalable learning analytics interventions tailored to teacher subgroups.
Equitable Group Work in Undergraduate Biology Courses: Leveraging a Complex Instruction Framework to Identify Pedagogical Strategies
www.lifescied.org/doi/10.1187/...
#Teaching #EdDev #STEMeducationEquitable Group Work in Underg...Equitable Group Work in Undergraduate Biology Courses: Leveraging a Complex Instruction Framework to Identify Pedagogical Strategies
https://www.lifescied.org/doi/10.1187/cbe.25-07-0154#Teaching #EdDev #STEMeducation
ABC Learning Design
An accessible guide to ABC Learning Design, showcasing its rapid, collaborative method and global adaptations that support innovative, flexible curriculum design.
Lia
LiaScript is a service for running free and interactive online courses, build with its own Markup-language. So check out the following course ;-)
Lia
LiaScript is a service for running free and interactive online courses, build with its own Markup-language. So check out the following course ;-)
Designing for authentic assessment: a scoping review - Higher Education
Authentic assessment has been adopted in higher education for decades, and its values in promoting learning and enhancing students’ employability hav
Summary of efforts to reform how college teaching is evaluated
engagedlearningcollective.substack.com/p/a-practica...
See TEval for some best practices:
teval.net
But also these references on bias in student evaluations of teaching:
docs.google.com/document/d/1...
#EdDev #Teaching #HigherEdA practical guide to modern te...
A practical guide to modern teaching evaluation
Dozens of institutions are piloting new ways to evaluate college teaching beyond student surveys. Here are the six steps they’re taking to fix a broken system.
A practical guide to modern teaching evaluation
Dozens of institutions are piloting new ways to evaluate college teaching beyond student surveys. Here are the six steps they’re taking to fix a broken system.
Doug Holton
Doug Holton