Effectiveness of longitudinal faculty development programs on MCQs items writing skills: A follow-up study

PLOS ONE, Nov 2019

This study examines the long-term impact of the faculty development programs on the multiple choice question (MCQ) items’ quality leading to study its effect on the students’ overall competency level during their yearly academic assessment. A series of longitudinal highly constructed faculty development workshops were conducted to improve the quality of the MCQs items writing skills. A total of 2207 MCQs were constructed by 58 participants for the assessment of 882 students’ cognitive competency level during the academic years 2012–2015. The MCQs were analyzed for the difficulty index (P-value), discriminating index (DI), presence/absence of item writing flaws (IWFs), and non-functioning distractors (NFDs), Bloom’s taxonomy cognitive levels, test reliability, and the rate of students’ scoring. Significant improvement in the difficulty index and DI were noticed during each successive academic year. Easy and poor discriminating questions, NFDs and IWFs were decreased significantly, whereas distractor efficiency (DE) mean score and high cognitive level (K2) questions were increased substantially during the each successive academic year. Improved MCQs’ quality leaded to increased competency level of the borderline students. Overall, the longitudinal faculty development workshops help in improving the quality of the MCQs items writing skills of the faculty that leads to students’ high competency levels.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0185895&type=printable

Effectiveness of longitudinal faculty development programs on MCQs items writing skills: A follow-up study

October Effectiveness of longitudinal faculty development programs on MCQs items writing skills: A follow-up study Hamza Mohammad Abdulghani 0 1 Mohammad Irshad 1 Shafiul Haque 1 Tauseef Ahmad 0 1 Kamran Sattar 0 1 Mahmoud Salah Khalil 0 1 0 Department of Medical Education, College of Medicine, King Saud University , Riyadh , Saudi Arabia , 2 Oral Microbiology Laboratory, Department of Bioclinical Sciences, Faculty of Dentistry, Health Sciences Centre, Kuwait University, Kuwait, 3 Research and Scientific Studies Unit, College of Nursing and Allied Health Sciences, Jazan University , Jazan , Saudi Arabia 1 Editor: Emmanuel Manalo, Kyoto University , JAPAN This study examines the long-term impact of the faculty development programs on the multiple choice question (MCQ) items' quality leading to study its effect on the students' overall competency level during their yearly academic assessment. A series of longitudinal highly constructed faculty development workshops were conducted to improve the quality of the MCQs items writing skills. A total of 2207 MCQs were constructed by 58 participants for the assessment of 882 students' cognitive competency level during the academic years 2012± 2015. The MCQs were analyzed for the difficulty index (P-value), discriminating index (DI), presence/absence of item writing flaws (IWFs), and non-functioning distractors (NFDs), Bloom's taxonomy cognitive levels, test reliability, and the rate of students' scoring. Significant improvement in the difficulty index and DI were noticed during each successive academic year. Easy and poor discriminating questions, NFDs and IWFs were decreased significantly, whereas distractor efficiency (DE) mean score and high cognitive level (K2) questions were increased substantially during the each successive academic year. Improved MCQs' quality leaded to increased competency level of the borderline students. Overall, the longitudinal faculty development workshops help in improving the quality of the MCQs items writing skills of the faculty that leads to students' high competency levels. - Data Availability Statement: All relevant data are within the paper and its Supporting Information files. Funding: The financial support for this study was provided by the COM, Research Centre, Deanship of Scientific Research, KSU, Riyadh, Saudi Arabia. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors have declared that no competing interests exist. Introduction Effective delivery of medical science education demands precise and efficient assessment system(s) capable of examining students' essential subjective knowledge, learning attitude and practical skills at high competency level. The assessment of high level competency changes students' learning behavior and ingrains enthusiasm in the students to accumulate subject's explicit and implicit information from the lecturers or other available resources [1±3]. In the assessment system, the use of multiple choice questions (MCQs) is a very common and well accepted method of evaluating diverse characteristics of the professional competencies of medical science education [ 4 ]. Moreover, medical education agencies are adopting the use of MCQs in the assessment system due to several advantages over the other assessment formats [ 5,6 ]. Most importantly, MCQs based assessment system is being used to test the cognitive competencies of a large number of students at-a-time with a broad range of curriculum content along with its objectivity, higher reliability, validity and ease of scoring [ 7,8 ]. Earlier studies have reported that MCQs are superior and apposite competency test for evaluating the subject's knowledge, comprehension, and can be designed to examine applications and analyses [ 9 ]. Construction of high quality MCQs can test advanced level critical thinking and application of the knowledge by evaluating the examinee's ability to integrate, synthesize and evaluate medical information [10±12]. Use of high quality MCQs in the assessment process determines the deep learning approach(s) of the students towards higher Bloom's taxonomical level of cognitive abilities such as interpretation, synthesis and application of the acquired knowledge instead of testing the recalling of the isolated facts [ 13 ]. Construction of high quality MCQ items is a very challenging task for the faculty members, especially for those who have never undergone precise and dedicated training [ 14,15 ]. However, MCQ items writing practice guidance has been well documented in the literature since long to guide the faculties who wish to construct MCQ items meeting with the standard assessment format [16±18]. Regardless of the availability of MCQ items writing guidelines, many reports have been published stating about various deficiencies and flaws in MCQ items of medical tests [17±20]. Deviations from MCQ items (...truncated)


This is a preview of a remote PDF: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0185895&type=printable

Hamza Mohammad Abdulghani, Mohammad Irshad, Shafiul Haque, Tauseef Ahmad, Kamran Sattar, Mahmoud Salah Khalil. Effectiveness of longitudinal faculty development programs on MCQs items writing skills: A follow-up study, PLOS ONE, 2017, Volume 12, Issue 10, DOI: 10.1371/journal.pone.0185895