1. Home
  2. Vol 4, No 1 (2023)
  3. Prenga

Enhancing Concept Inventory Analysis by Using Indexes, Optimal Histogram Idea, and the Likert Analysis

Dode Prenga, Elmira Kushta, Fatjon Musli

Abstract


Since the introduction of the Force Concept Inventory (FCI) in 1992, the CI tests have been widely used for measuring conceptual knowledge and for studying teaching issues in almost all disciplines and levels of study. A standard concept inventory analysis includes the design of a qualitative test, adequate realization of testing, calibration procedure, and comprehensive analysis of its findings. Usually, the CI test calibration is carried out through the Rasch sociometric technique, which is also used for calculating crucial indicators of knowledge such as item difficulties, students’ abilities, and many more. Whereas the quality of the tests’ design can be guaranteed by using certified and professional CI tests, the statistical adequacy of the testing merits critical attention before going on to the final step of the analysis. Also, the analysis of CI outcomes can be advanced by contemplating auxiliary tools and complementary techniques. In this framework, we propose to enforce the test index validity requirement for qualifying the CI outcomes as local or global. Specifically, the conclusions of CI analysis are acceptable for the whole population from which the sample has been extracted if the test's indexes comply with the validity requirements provided by the index theory. In the case when test indexes are out of validity range and re-conducting them is impractical for some objective circumstances or research specifics, we suggest injecting some new records into the existing one or mixing the data gathered from different samples until the new indexes are in the desired range. Using this methodology, we have reviewed our previous FCI tests, which were initially intended to demonstrate the impairment of learning in the physics discipline triggered by online learning during the pandemic closure. Through this renormalization procedure, we obtained a credible assessment of the understanding of mechanics and electromagnetism in high school students who followed online lectures during the pandemic closure. Also, by using indexes’ validity as an auxiliary tool, we identified that for measuring the knowledge of electromagnetism in students enrolled in branches where physics is a basic discipline, a shortened version of the BEMA test was a better instrument than the corresponding shortened EMCI test. Next, we used the optimal histogram idea borrowed from distribution fitting procedures to identify the natural levels of students’ abilities for solving a certain CI test. Another intriguing proposal presented in this work consists of combining an ad-hoc Likert scale assignment for usual errors in physics exams with the FCI designation of the basic commonsense confusion in mechanics for identifying their pairing features in common exams. We believe that the methods proposed herein can improve CI analysis in more general senses.

 

Doi: 10.28991/HEF-2023-04-01-08

Full Text: PDF


Keywords


Concept Inventory; Physics Knowledge, The Rasch Model; Likert Scale.

References


Crooks, N. M., & Alibali, M. W. (2014). Defining and measuring conceptual knowledge in mathematics. Developmental Review, 34(4), 344–377. doi:10.1016/j.dr.2014.10.001.

Savinainen, A., & Scott, P. (2002). The force concept inventory: A tool for monitoring student learning. Physics Education, 37(1), 45–52. doi:10.1088/0031-9120/37/1/306.

Rahmawati, Rustaman, N. Y., Hamidah, I., & Rusdiana, D. (2018). The development and validation of conceptual knowledge test to evaluate conceptual knowledge of physics prospective teachers on electricity and magnetism topic. Jurnal Pendidikan IPA Indonesia, 7(4), 483–490. doi:10.15294/jpii.v7i4.13490.

Savinainen, A., & Viiri, J. (2008). The force concept inventory as a measure of student’s conceptual coherence. International Journal of Science and Mathematics Education, 6(4), 719–740. doi:10.1007/s10763-007-9103-x.

O’Shea, A., Breen, S., & Jaworski, B. (2016). The Development of a Function Concept Inventory. International Journal of Research in Undergraduate Mathematics Education, 2(3), 279–296. doi:10.1007/s40753-016-0030-5.

Luangrath, P., Pettersson, S., & Benckert, S. (2011). On the use of two versions of the force concept inventory to test conceptual understanding of mechanics in Lao PDR. Eurasia Journal of Mathematics, Science and Technology Education, 7(2), 103–114. doi:10.12973/ejmste/75184.

Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force Concept Inventory. The Physics Teacher, 30(3), 141–158. doi:10.1119/1.2343497.

Sands, D., Parker, M., Hedgeland, H., Jordan, S., & Galloway, R. (2018). Using concept inventories to measure understanding. Higher Education Pedagogies, 3(1), 173–182. doi:10.1080/23752696.2018.1433546.

Khairandy, R., Barkatullah, A. H., Huda, M. K., & Amir, A. Y. (2022). Exploring Social Contracts: Enhancing Cooperation and Collaboration between Businesses and Communities. Journal of Human, Earth, and Future, 3(4), 452-460. doi:10.28991/HEF-2022-03-04-005.

Hambleton, R. K., & Swaminathan, H. (1985). Item Response Theory. Springer, Dordrecht, Netherlands. doi:10.1007/978-94-017-1988-9.

Embretson, S. E., & Reise, S. P. (2013). Item response theory. Psychology Press, London, United Kingdom. doi:10.4324/9781410605269.

van der Linden, W. J., & Hambleton, R. K. (1997). Handbook of Modern Item Response Theory. Springer, New York, United States. doi:10.1007/978-1-4757-2691-6.

Rasch, G. (1961, January). On general laws and the meaning of measurement in psychology. Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, 1 January, 1961, Berkeley, United States.

Klymkowsky, M. W., & Garvin-Doxas, K. (2020). Concept Inventories: Design, Application, Uses, Limitations, and Next Steps. Active Learning in College Science. Springer, Cham, Switzerland. doi:10.1007/978-3-030-33600-4_48.

McCombes, S. (2023). Sampling Methods | Types, Techniques & Examples. Scribbr, Amsterdam, Netherlands. Available online: https://www.scribbr.com/methodology/sampling-methods/ (accessed on January 2023).

Handhika, J., Huriawati, F., & Fitriani, N. (2017). Force concept inventory (FCI) representation of high school students (SMA & MA). Journal of Physics: Theories and Applications, 1(1), 29. doi:10.20961/jphystheor-appl.v1i1.4706.

Smaill, C., & Rowe, G. (2012). Electromagnetics Misconceptions: How Common Are These Amongst First- and Second-year Electrical Engineering Students? 2012 ASEE Annual Conference & Exposition Proceedings. doi:10.18260/1-2--21268.

Raduta, C. (2005). General students' misconceptions related to Electricity and Magnetism. arXiv preprint, physics/0503132. doi:10.48550/arXiv.physics/0503132.

McColgan, M. W., Finn, R. A., Broder, D. L., & Hassel, G. E. (2017). Assessing students’ conceptual knowledge of electricity and magnetism. Physical Review Physics Education Research, 13(2), 20121. doi:10.1103/PhysRevPhysEducRes.13.020121.

Laverty, J. T., & Caballero, M. D. (2018). Analysis of the most common concept inventories in physics: What are we assessing? Physical Review Physics Education Research, 14(1), 10123. doi:10.1103/PhysRevPhysEducRes.14.010123.

Prenga, D., Kushta, E., Peqini, K., Osmani, R., & Hysenlli, M. (2023). Analyzing influential factors on physics knowledge weakness in high school students due to the pandemic-imposed online learning and a discussion for enhancing strategies. AIP Conference Proceedings, vol. 2872, no. 1. doi:10.1063/5.0162933.

Kushta, E., Dode Prenga, S. M., & Dhoqina, P. (2022). Assessment of the Effects of Compulsory Online Learning During Pandemic Time on Conceptual Knowledge Physics. Mathematical Statistician and Engineering Applications, 71(4), 6382-6391. doi:10.17762/msea.v71i4.1228.

Pattanasing, K., Aujirapongpan, S., Dowpiset, K., Chanthawong, A., Jiraphanumes, K., & Hareebin, Y. (2022). Dynamic Knowledge Management Capabilities: An Approach to High-Performance Organization. HighTech and Innovation Journal, 3(3), 243-251. doi:10.28991/HIJ-2022-03-03-01.

Ding, L., Chabay, R., Sherwood, B., & Beichner, R. (2006). Evaluating an electricity and magnetism assessment tool: Brief electricity and magnetism assessment. Physical Review Special Topics - Physics Education Research, 2(1), 10105. doi:10.1103/PhysRevSTPER.2.010105.

Aubrecht, G. J., & Aubrecht, J. D. (1983). Constructing objective tests. American Journal of Physics, 51(7), 613–620. doi:10.1119/1.13186.

Notaros, B. M. (2002). Concept inventory assessment instruments for electromagnetics education. IEEE Antennas and Propagation Society International Symposium (IEEE Cat. No.02CH37313). doi:10.1109/aps.2002.1016436.

Hansen, J., & Stewart, J. (2021). Multidimensional item response theory and the Brief Electricity and Magnetism Assessment. Physical Review Physics Education Research, 17(2), 20139. doi:10.1103/PhysRevPhysEducRes.17.020139.

Kožuchová, M., Barnová, S., & Stebila, J. (2022). Inquiry as a part of educational reality in technical education. Emerging Science Journal, 6 (Special issue), 225-240. doi:10.28991/ESJ-2022-SIED-016.

Linacre, J. M. (2020). Fit diagnosis: Infit outfit mean-square standardized, Winsteps. Available online: https://www. winsteps.com/winman/misfitdiagnosis.htm (accessed on February 2023).

Zaiontz, C. (2023). Building a Rasch Model. Real Statistics Using Excel. Available online: https://real-statistics.com/reliability/ item-response-theory/building-rasch-model/ (accessed on February 2023).

Anderson, C. J., Verkuilen, J., & Peyton, B. L. (2010). Modeling Polytomous Item Responses Using Simultaneously Estimated Multinomial Logistic Regression Models. Journal of Educational and Behavioral Statistics, 35(4), 422–452. doi:10.3102/1076998609353117.

Planinic, M. (2006). Assessment of difficulties of some conceptual areas from electricity and magnetism using the Conceptual Survey of Electricity and Magnetism. American Journal of Physics, 74(12), 1143–1148. doi:10.1119/1.2366733.

Planinic, M., Boone, W. J., Susac, A., & Ivanjek, L. (2019). Rasch analysis in physics education research: Why measurement matters. Physical Review Physics Education Research, 15(2). doi:10.1103/PhysRevPhysEducRes.15.020111.

Bevans, R. (2022, December 05). Choosing the Right Statistical Test | Types & Examples. Scribbr, Amsterdam, Netherlands. Available online: https://www.scribbr.com/statistics/statistical-tests/ (accessed on February 2023).

Bruning, J. L., & Kintz, B. L. (1987). Computational handbook of statistics (3rd Ed.). Foresman and Company, Northbrook, United States.

Martínez-Mesa, J., González-Chica, D. A., Duquia, R. P., Bonamigo, R. R., & Bastos, J. L. (2016). Sampling: How to select participants in my research study? Anais Brasileiros de Dermatologia, 91(3), 326–330. doi:10.1590/abd1806-4841.20165254.

Chabay, R. (1997). Qualitative Understanding and Retention. AAPT Announcer, 27(2), 96.

Liu, X. (2010). Using and developing measurement instruments in science education: A Rasch modeling approach. Information Age Pub, Charlotte, United States.

Coletta, V. P., & Phillips, J. A. (2005). Interpreting FCI scores: Normalized gain, preinstruction scores, and scientific reasoning ability. American Journal of Physics, 73(12), 1172–1182. doi:10.1119/1.2117109

Boone, W. J. (2016). Rasch analysis for instrument development: Why, when, and how? CBE Life Sciences Education, 15(4). doi:10.1187/cbe.16-04-0148.

Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 44–53.

Planinic, M., Ivanjek, L., & Susac, A. (2010). Rasch model based analysis of the Force Concept Inventory. Physical Review Special Topics - Physics Education Research, 6(1), 1-11. doi:10.1103/physrevstper.6.010103.

Granger, C. (2008). Rasch analysis is important to understand and use for measurement. Rasch Measurement Transactions, 21(3), 1122-1123. Available online: https://www.rasch.org/rmt/rmt213d.htm (accessed on January 2023).

Umarov, S., Tsallis, C., & Steinberg, S. (2008). On a q-central limit theorem consistent with nonextensive statistical mechanics. Milan Journal of Mathematics, 76(1), 307–328. doi:10.1007/s00032-008-0087-y.

Popp, S. E. O., & Jackson, J. C. (2009). Can assessment of student conceptions of force be enhanced through linguistic simplification? A Rasch model common person equating of the FCI and the SFCI. Annual Meeting of the American Educational Research Association, April, 2009, San Diego, United States.

39-Stoen, S. M., McDaniel, M. A., Frey, R. F., Hynes, K. M., & Cahill, M. J. (2020). Force concept inventory: More than just conceptual understanding. Physical Review Physics Education Research, 16(1), 10105. doi:10.1103/PhysRevPhysEducRes.16.010105.

MathWorks (2023). MATLAB Online. Available online: https://www.mathworks.com/products/matlab-online.html (accessed on February 2023).

Knuth, K. H. (2019). Optimal data-based binning for histograms and histogram-based probability density models. Digital Signal Processing, 95, 102581. doi:10.1016/j.dsp.2019.102581.

Scott, D. W. (1979). On optimal and data-based histograms. Biometrika, 66(3), 605–610. doi:10.2307/2335182.

Scott, D. W. (2015). Multivariate density estimation: theory, practice, and visualization. John Wiley & Sons, Hoboken, United States. doi:10.1002/9781118575574.

Freedman, D., & Diaconis, P. (1981). On the histogram as a density estimator: L2 theory. Zeitschrift Für Wahrscheinlichkeitstheorie Und Verwandte Gebiete, 57(4), 453–476. doi:10.1007/BF01025868.

Nyutu, E. N., Cobern, W. W., & Pleasants, B. A. S. (2021). Correlational study of student perceptions of their undergraduate laboratory environment with respect to gender and major. International Journal of Education in Mathematics, Science and Technology, 9(1), 83–102. doi:10.46328/ijemst.1182.

Gliem, J. a, & Gliem, R. R. (2003). Calculating, Interpreting, and Reporting Cronbach’s Alpha Reliability Coefficient for Likert-Type Scales. 2003 Midwest Research to Practice Conference in Adult, Continuing, and Community Education, 1992, 82–88. doi:10.1109/PROC.1975.9792.

52-Louangrath, P. I., & Sutanapong, C. (2018). Validity and reliability of survey scales. International Journal of Research & Methodology in Social Science, 4(3), 99-114.


Full Text: PDF

DOI: 10.28991/HEF-2023-04-01-08

Refbacks

  • There are currently no refbacks.


Copyright (c) 2023 Dode Prenga, Elmira Kushta, Fatjon Musli