Examining the Factors Behind Experimental Group Superiority in Quasi-Experimental Research: A Mixed-Methods Analysis of MA Theses
Keywords:
Quasi-experimental design; Sultan Qaboos University; Experimental group; Control groupAbstract
When random assignment is not feasible in education research, quasi-experimental designs are used to examine the impact of instructional interventions on other variables. Notably, a recurring pattern in such studies is the outperformance of the experimental groups over the control groups, which raises questions about the validity and interpretation of these outcomes. This study investigated the factors that lead to the consistent outperformance of experimental groups over control groups in quasi-experimental research. By adopting a mixed-methods approach, the researchers analysed 100 MA theses of the Department of Curriculum and Instruction at Sultan Qaboos University in the Sultanate of Oman. This analysis was supported by a survey that was distributed to lecturers at four universities in the country to elicit their insights on issues that include selection bias, intervention effectiveness, implementation differences, and the Hawthorne effect, as potential contributors to this trend. Based on the survey responses of the lecturers, the primary contributing factor to the superior performance of experimental groups compared to control groups was the Hawthorne effect, followed by the effectiveness of the intervention and selection bias as secondary factors, accounting for the observed trend. The findings offer a comprehensive explanation of the underlying causes and present practical recommendations for improving the design and interpretation of quasi-experimental studies in education research.
https://doi.org/10.26803/ijlter.24.9.1
References
Andrade, C. (2021). The limitations of quasi-experimental studies, and methods for data analysis when a quasi-experimental research design is unavoidable. Indian Journal of Psychological Medicine, 43(5), 451–452.? https://doi.org/10.1177/02537176211034707
Barnett, A. G., Van der Pols, J. C., & Dobson, A. J. (2005). Regression to the mean: What it is and how to deal with it. International Journal of Epidemiology, 34(1), 215–220.? https://doi.org/10.1093/ije/dyh299
Biddle, N., & Curchin, K. (2024). Randomised controlled trials and quasi-experimental designs. In B. M. Smyth, M. A. Martin, & M. Downing (Eds.), The Routledge handbook of human research ethics and integrity in Australia (pp. 187–198). Routledge.? https://doi.org/10.4324/9781003319733-20
Biesta, G. (2010). “Why ‘what works’ still won’t work: From evidence-based education to value-based education. Studies in Philosophy and Education, 29, 491–503. https://doi.org/10.1007/s11217-010-9191-x
Century, J., Rudnick, M., & Freeman, C. (2010). A framework for measuring fidelity of implementation: A foundation for shared language and accumulation of knowledge. American Journal of Evaluation, 31(2), 199–218.? https://doi.org/10.1177/1098214010366173
Cook, B. G., & Cook, L. (2016). Research designs and special education research: Different designs address different questions. Learning Disabilities Research & Practice, 31(4), 190–198.? https://doi.org/10.1111/ldrp.12110
Cook, D. (1967). The impact of the Hawthorne effect on experimental designs in educational research. Final report No. BR-5-0725. Washington DC Bureau of Research.
Cook, T. D. & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Houghton Mifflin.
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. https://doi.org/10.1007/s10464-008-9165-0
El Soufi, N., & See, B. H. (2019). Does explicit teaching of critical thinking improve critical thinking skills of English language learners in higher education? A critical review of causal evidence. Studies In Educational Evaluation, 60, 140–162.? https://doi.org/10.1016/j.stueduc.2018.12.006
Gillespie, R. (1993). Manufacturing knowledge: A history of the Hawthorne experiments. Cambridge University Press.?
Harris, A. D., McGregor, J. C., Perencevich, E. N., Furuno, J. P., Zhu, J., Peterson, D. E., & Finkelstein, J. (2006). The use and interpretation of quasi-experimental studies in medical informatics. Journal of the American Medical Informatics Association, 13(1), 16?23.? https://doi.org/10.1197/jamia.M1749
Infante-Rivard, C., & Cusson, A. (2018). Reflection on modern methods: selection bias—a review of recent developments. International Journal of Epidemiology, 47(5), 1714?1722.? https://doi.org/10.1093/ije/dyy138
Jobson, D., Li, Y., Nishimura, N., Ohashi, K., Yang, J., & Matsumoto, T. (2024). Covariate ordered systematic sampling as an improvement to randomized controlled trials. Submitted to the 33rd ACM International Conference on Information and Knowledge Management, October. https://doi.org/10.1145/3627673.3679892
Kahan, B. C., Rehal, S., & Cro, S. (2015). Risk of selection bias in randomised trials. Trials, 16, 1–7.? http://dx.doi.org/10.1186/s13063-015-0920-x
Kazdin, A. E. (2019). Single-case research designs: Methods for clinical and applied settings. Oxford University Press.
Kelchtermans, G. (2009). Who I am in how I teach is the message: Self-understanding, vulnerability and reflection. Teachers and Teaching: Theory and Practice, 15, 257–272. http://dx.doi.org/10.1080/13540600902875332
Kim, Y., & Steiner, P. (2016). Quasi-experimental designs for causal inference. Educational Psychologist, 51(3–4), 395–405.? https://doi.org/10.1080/00461520.2016.1207177
Levitt, S. D., & List, J. A. (2011). Was there really a Hawthorne effect at the Hawthorne plant? An analysis of the original illumination experiments. American Economic Journal: Applied Economics, 3(1), 224–238. https://doi.org/10.1257/app.3.1.224
Maciejewski, M. L. (2018). Quasi-experimental design. Biostatistics & Epidemiology, 4(1), 38–47.? https://doi.org/10.1080/24709360.2018.1477468
Mansour, N. (2009). Science teachers’ beliefs and practices: Issues, implications and research agenda. International Journal of Environmental & Science Education, 4, 25–48.
Marsden, E., & Torgerson, C. J. (2012). Single group, pre-and post-test research designs: Some methodological concerns. Oxford Review of Education, 38(5), 583–616.? https://doi.org/10.1080/03054985.2012.731208
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615–631.? https://doi.org/10.1177/1745691612459058
Palmer, P. J. (2007). The courage to teach: Exploring the inner landscape of a teacher’s life. Jossey-Bass. http://dx.doi.org/10.1111/j.1744-1722.2000.tb00290.x
Reichardt, C. S. (2019). Quasi-experimentation: A guide to design and analysis. Guilford Publications.?
Richardson, J. T. E. (2005). Students’ approaches to learning and teachers’ approaches to teaching in higher education. Educational Psychology, 25, 673–680. http://dx.doi.org/10.1080/01443410500344720
Rosenbaum, P. R. (2010). Design of observational studies. Springer.?
Scher, L., Kisker, E., & Dynarski, M. (2015). Designing and conducting strong quasi-experiments in education (Version 2). Decision Information Resources.?
Seltman, H. J. (2013, December 3). Experimental design and analysis.? http://www.stat.cmu.edu/?hseltman/309/Book/Book.pdf
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Siedlecki, S. L. (2020). Quasi-experimental research designs. Clinical Nurse Specialist, 34(5), 198-202.? https://doi.org/10.1097/NUR.0000000000000540
Trochim, W. M. K. (2006). The qualitative debate: Research methods knowledge base. Conjointly. http://www.socialresearchmethods.net/kb/qualmeth.php
Trochim, W. M. K. (2020). Research methods: The essential knowledge base. Cengage Learning.
Vannan, K. (2021). History of the Hawthorne effect. https://doi.org/10.1002/9781119111931.ch47
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Sheikha Ali Al-Buraiki, Salama Saif Al-Siyabi, Ali Salim Alghafri
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
All articles published by IJLTER are licensed under a Creative Commons Attribution Non-Commercial No-Derivatives 4.0 International License (CCBY-NC-ND4.0).
