VFR-into-IMC: An Analysis of Two Training Protocols on Weather-Related Posttest Scores

Journal of Aviation Technology and Engineering, Oct 2017

According to the Aircraft Owners and Pilots Association Air Safety Institute, 264 accidents were identified as continued visual flight rules (VFR) into instrument meteorological conditions (IMC), during the past ten years. Approximately 89% of those VFR-into-IMC accidents were fatal, causing hundreds of deaths. VFR-into-IMC has been a major concern for the general aviation community, prompting focused efforts. Research, data analyses, outreach, training, and education are recommended practices to address risks associated with VFR-into-IMC. Researchers of the current study sought to evaluate the cause and effect relationship between two training protocols and weather-related posttest scores. A pretest–posttest experimental design was utilized at two testing locations. Participants were randomly assigned to one of three groups: a control group, an interactive online training group, or an interactive workshop group. An analysis of covariance was used to determine whether there was a significant difference between mean posttest scores among the experimental groups while controlling for pretest scores. The treatments did not appear to significantly increase posttest scores after controlling for pretest scores, at either experiment location. Though the results of this study did not yield anticipated findings, much was learned and potentially helpful to general aviation researchers seeking to mitigate VFR-into-IMC encounters. Recommendations for future research and practices are discussed.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

http://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1150&context=jate

VFR-into-IMC: An Analysis of Two Training Protocols on Weather-Related Posttest Scores

Journal of Aviation Technology and Engineering VFR-into-IMC: An Analysis of Two Training Protocols on Weather-Related Posttest Scores Julius C. Keller Thomas Carney Allen Xie Wesley Major Matt Price According to the Aircraft Owners and Pilots Association Air Safety Institute, 264 accidents were identified as continued visual flight rules (VFR) into instrument meteorological conditions (IMC), during the past ten years. Approximately 89% of those VFR-into-IMC accidents were fatal, causing hundreds of deaths. VFR-into-IMC has been a major concern for the general aviation community, prompting focused efforts. Research, data analyses, outreach, training, and education are recommended practices to address risks associated with VFR-into-IMC. Researchers of the current study sought to evaluate the cause and effect relationship between two training protocols and weather-related posttest scores. A pretest-posttest experimental design was utilized at two testing locations. Participants were randomly assigned to one of three groups: a control group, an interactive online training group, or an interactive workshop group. An analysis of covariance was used to determine whether there was a significant difference between mean posttest scores among the experimental groups while controlling for pretest scores. The treatments did not appear to significantly increase posttest scores after controlling for pretest scores, at either experiment location. Though the results of this study did not yield anticipated findings, much was learned and potentially helpful to general aviation researchers seeking to mitigate VFR-into-IMC encounters. Recommendations for future research and practices are discussed. VFR-into-IMC; general aviation; weather; aeronautical; decision-making; pilot training Introduction According to a National Transportation Safety Board (NTSB) accident report (NTSB, 2012) , on July 11, 2012, a Cirrus SR20 impacted terrain and the pilot (the sole occupant) was fatally injured. The accident investigation revealed the noninstrument rated pilot, who had approximately 340 total flight hours, departed without filing a flight plan from Millington, TN, intending to fly to Pensacola, FL. There was no indication the pilot checked the weather. A person who spoke with the accident pilot prior to his departure reported the pilot was in a hurry due to adverse weather in the area. Meteorological information at the time and place of departure indicated variable wind at four knots, six statute miles visibility with mist, overcast ceiling at 900 feet, temperature 24˚C, dew point 22˚C, and an altimeter setting of 29.99 in. Hg. Infrared satellite images showed a large area of low stratiform clouds with tops near 8000 feet along the route of flight. Additionally, rain showers were reported east of the accident site (NTSB, 2012) . According to the accident report, a review of radar return data indicated the accident airplane flew at a relatively consistent altitude of about 1000 feet mean sea level (MSL) for approximately 25 minutes after takeoff (NTSB, 2012) . Primary and multifunction display data were not recovered from the aircraft wreckage, due to excessive damage. However, enhanced ground proximity warning system data were retrieved. The data analysis revealed that, during the last 10 seconds, the rate of descent increased rapidly from 5000 feet per minute to 15,000 feet per minute. Additionally, the bank angle varied from a 24 degree roll to the left to a 28 degree roll to the right during the last 10 seconds. No anomalies regarding the aircraft or pilot were found during the investigation (NTSB, 2012) . The accident pilot was 48 years old and held a private pilot certificate with an airplane single-engine land rating. His medical certificate had no limitations. The probable cause of the accident was attributed to the non-instrument rated pilot’s decision to continue into instrument meteorological conditions (IMC), which resulted in spatial disorientation, loss of control, and subsequent impact into terrain (NTSB, 2012) . This accident summary highlights many factors associated with continued visual flight rules (VFR) into IMC. Though it is impossible to conclusively assert the reasons for the accident, evidence indicates there were shortcomings in the pilot’s knowledge, skills, and abilities. According to the Aircraft Owners and Pilots Association Air Safety Institute (2015), between 2005 and 2015, 264 accidents were identified as continued VFR-into-IMC. Although the occurrence of this accident type is relatively low when compared to the approximately 1200 general aviation (GA) accidents that occur each year, historically VFR-into-IMC accidents have had an approximate 90% fatality rate, resulting in hundreds of deaths. Primarily due to the high fatality rate, VFR-into-IMC has been a major concern for the GA community, prompting focused efforts. The Federal Aviation Administration (FAA) is committed to reducing GA’s fatal accident rate by 10 percent over the next ten-year period (FAA, 2015). The FAA plan of action for improving safety includes: data analysis, outreach and education, flight instructor training, collaboration with industry, and establishing committees to develop interventions based on research (FAA, 2015). The study that is the basis for this paper was a continuation (Phase II) of FAA-sponsored research to investigate gaps in pilot training and knowledge as a part of Project 4 (Weather Technology in the Cockpit) of the Partnership to Enhance General Aviation Safety, Accessibility, and Sustainability (PEGASAS). PEGASAS is the current FAA Center of Excellence for General Aviation. During the Phase I study of PEGASAS Project 4, researchers distributed a national survey to GA pilots. Gaps in pilot knowledge that pertain to VFR-into-IMC, identified during Phase I, included: (1) the inability of pilots to correlate, interpret, and apply weather information related to VFRinto-IMC; (2) a perceived gap in skills related to VFRinto-IMC decision-making; and (3) pilots’ inadequate retention of weather knowledge (Carney et al., 2015) . The purpose of the current study was to evaluate the effectiveness of two VFR-into-IMC training protocols, through a pretest–posttest experimental design. One training protocol utilized an interactive online training module, while the other incorporated a conventional pilot training workshop. In addition to the two training protocols, a control group was included as part of the research design. Data were collected at two locations, as further discussed in the methodology section. The researchers expected that either one or both intervention groups would have significantly higher posttest scores than the control group. This action research gives the GA community a clearer understanding of the complexities related to VFR-into-IMC training and risk mitigation. The next section discusses extant VFR-into-IMC research and justification for the instructional methods used in this study. Literature Review Extant VFR-into-IMC Research Typically, aviation incidents and accidents are caused by multiple factors. Several research studies point to inadequate training as a factor that may lead to a VFR-into-IMC encounter. According to Ison (2014) , studies regarding weather and VFR-into-IMC have found that pilots often misunderstand weather, and/or do not acquire sufficient training on how to use weather reports. Ison (2014) established: Considering that pilots who were involved in continued VFR-into-IMC received a weather briefing, it points to the need to focus on weather education as well as hazardous pilot attitudes. Perhaps pilots are not heeding the warnings of weather briefers or sources either because they do not grasp the required knowledge to evaluate weather reports and factors (or) they may feel as though they are invulnerable due to having made the flight before, overconfidence, or other negative attitudes. (p. 22) Coyne, Baldwin, and Latorella (2008) asserted that pilots found weather knowledge and the related decision-making process to be the most challenging during flight training. Pilots often felt there was a lack of support in this subject area and were left on their own to gain a deeper understanding of weather-related information. Additionally, the flight training environment may have more restrictive weather minimums, reducing pilots’ ability to experience weather phenomena. Moreover, different regions of the United States may limit opportunities to safely gain weather experience. It is also possible that outside of the flight training environment the lack of supervision from a flight instructor may increase the risk of VFR-into-IMC encounters. Knect and Lenz (2010) contacted and interviewed 100 pilots who submitted VFR-into-IMC reports to the Aviation Safety Reporting System. Twenty-seven of the pilots were non-instrumented rated and encountered IMC inadvertently. After a thorough review of qualitative and quantitative data, the researchers concluded these pilots had the least amount of training, aeronautical knowledge, and experience. Additionally, the non-instrument rated pilots flew less capable aircraft, in terms of weather technology. Based on the results, the authors suggested a need for training in the areas of risk perception and strategies for avoiding/handling an IMC encounter. Wiggins and O’Hare (2003) evaluated pilot time required to develop alternative plans of action when handling adverse weather conditions. Results indicated that the less experience a pilot had, the longer it took to develop an alternative course of action. Follow-up research indicated cue-based training could improve time to make weatherrelated decisions (Wiggins & O’Hare, 2003) . Hunter, Martinussen, Wiggins, and O’Hare (2011) investigated personal characteristics of pilots who survived VFR-into-IMC encounters using pilot judgment, hazardous events, and personal minimums scales. Demographic and flight experience data were also collected during the study. Usable data were collected from 364 participants. Participants were categorized into three groups: 144 who indicated they experienced flying into weather (both deliberately and inadvertently), 114 who had conducted a flight in which weather conditions were a concern, and 106 who reported no flights for which weather was entered or was a major concern. The two former groups were used for primary analyses. Notable results from the study led to a discussion pertinent to pilot training. The authors suggested an emphasis of educating pilots on the importance of using a weather briefer, teaching them how to develop conservative personal minimums, and developing more detailed alternative plans. Previous research indicated the need for better training to mitigate the risks of VFR-into-IMC encounters. Two primary aspects come into play when attempting to enhance current training practices: (1) selection of topics to include in the training and (2) consideration of various instructional delivery platforms. Based on an evaluation of previous literature, two major topics arise: weather information and aeronautical decision-making pertaining to VFR-into-IMC. The next section explores research pertaining to instructional methods. Justification for Selected Instructional Methods Researchers of the current study utilized two training protocols. One was an online module and the second was a workshop. The two instructional methods were selected because of promising potential or the need for further research based on extant literature. A number of researchers have investigated the utilization of online training courses, due to their increasingly widespread use. Wisher and Olson (2003) examined articles from an online database to determine the effectiveness of web-based training modules on learning. Forty-seven articles were used for the study. Fifteen of the articles reported effect sizes when comparing online courses to traditional classroom presentations. Results indicated the average effect size was 0.24, meaning the average posttest score increased 10 percentage points. However, a limitation to the study was the small sample size, with large variability. Effect sizes ranged from 20.4 to 1.6. Based on evidence provided by Wisher and Olson (2003) , computer-based learning may lead to an improvement in learning effectiveness, when compared to traditional classroom presentations. The study reports broad categories of the fields of study, and did not report the complexity of the topics taught. Sitzmann, Kraiger, Stewart, and Wisher (2006), in an examination of research articles, determined that the effectiveness of online courses depends on the material taught. Ninety-six research articles produced data from 19,331 trainees enrolled in 168 courses. The subject matter included psychology, engineering, computer programming, business, and technical writing. Participants in the studies were employees and undergraduate and graduate students. Overall results indicated web-based instruction was six percent more effective than traditional classroom instruction, when teaching declarative knowledge. When web-based instruction was used to supplement face-to-face instruction, results indicated a higher level of effectiveness in both declarative and procedural knowledge. However, web-based training was not more effective at teaching procedural knowledge. With regard to aviation, few research studies have examined the effectiveness of interactive online training modules pertaining to weather. Knecht, Ball, and Lenz (2010) evaluated video training products pertaining to aviation weather-related knowledge and flight performance during deteriorating conditions. Fifty general aviation pilots participated in the study. Participants were assigned to one of two groups. The first group watched a 90-minute video that did not pertain to weather. The other group watched a 90-minute weather training video. Pretests and posttests were administered. The researchers concluded that the training videos were not effective, due to the complexity of weather and decision-making (Knecht et al., 2010) . It may also be possible that the lack of interaction while participants watched the video contributed to its ineffectiveness. Researchers of the current study chose to use an interactive online module to provide additional evidence pertaining to the effectiveness of this instructional method. A workshop was the second instructional method used in this study. Surprisingly, no research studies that evaluated the effectiveness of VFR-into-IMC workshops have been reported in the literature. Therefore, the literature from other fields was examined. According to Brooks-Harris and Stock-Ward (1999) , a workshop is typically a highly interactive session facilitated by an expert. ‘‘Workshop’’ and ‘‘seminar’’ are words often interchanged. Similar to a workshop, a seminar can last from a half-day to two days and is typically facilitated by an expert. The distinguishing difference is that a workshop is more interactive. There is communication among workshop participants and facilitator, while a seminar typically has one-way communications, from facilitator to participants. Furthermore, a workshop tends to have interactive assignments embedded into the program. Learners are often enticed by face-to-face presentation of training material and exchanges among participants. In addition, instructional methods utilized during a workshop can be wide-ranging. Mixtures of instructional methods may improve student understanding of a subject, improve communication, and positively affect different learning styles. The execution of the appropriate instructional methods may make the education process more effective (Guskey, 2014) . A study conducted by Rust (1998) examined pre- and post-workshop surveys directed towards faculty members. The primary purpose of the study was to determine whether participant attitudes towards teaching practices would change. Workshop topics included teaching large classes, assessments, curriculum, supervising post graduates, problem-based learning, and teaching in higher education. Instructional methods included interactive components. Five hundred faculty members responded to the survey. Results provided evidence that workshops may promote change in participants, provide encouragement, and increase confidence in using desired teaching methods. Occupational health researchers conducted a study to evaluate interactive fatigue management workshops for nurses (Ali, Chalder, & Madan, 2014) . Research survey items asked participants how confident they were at discussing, diagnosing, and managing chronic fatigue. Additionally, participants were asked how satisfied they were with the workshop. The questionnaires were distributed prior to, directly afterwards, and four months after attendance. Seventy-three participants completed all three questionnaires. Results supported the concept that knowledge may be improved by interactive workshops. Eighty-nine percent of participants rated their experience between five and seven on a seven-point Likert scale (Ali et al., 2014) . Empirical evidence from multiple fields has shown that workshops may be effective at changing knowledge, skills, abilities, and attitudes. Previous research has shown that well-planned workshops with lectures, reflective thinking, discussions, and case studies may be conducive for deeper learning. However, research pertaining to workshops and complex aviation topics is still lacking. Research pertaining to online training modules exhibits mixed results, particularly with complex subject matter. It was the intent of the researchers in this study to provide additional evidence related to the effectiveness of online training modules, and to compare online effectiveness with that of a conventional workshop. The study sought to yield evidence to assist researchers in the field to sort through the complexities and continue to find beneficial mitigation strategies. Methodology This study examined the effectiveness of two training protocols on posttest scores at two separate locations, the FAA William J. Hughes Technical Center (WJHTC) near Atlantic City, NJ (Location 1), and Purdue University Airport at West Lafayette, IN (Location 2). The researchers utilized a pretest–posttest experimental design. All participants completed intake procedures, which included a briefing, review and signing of consent forms, completing a pretest, and random assignment to one of three groups for the research. Participants assigned to the control group received no weather training intervention prior to flying two scenarios in a flight training device. Participants assigned to the interactive online group were asked to independently complete an online training module, which typically took an hour to complete. Participants assigned to the workshop group were asked to engage in an approximately two-hour workshop facilitated by the principal investigator, who is a professional pilot, flight instructor, and expert in aviation meteorology. Once the interventions (if assigned) were completed, participants completed a flight training device session, which lasted approximately one hour. Keller (2015) and Carney et al. (2015) discussed flight training device session results in separate reports. After completion of the flight training device sessions, all participants were asked to complete the posttest. At Location 1, participants completed the pretest, flight training device sessions, and posttest within the same day. Participants at Location 2 completed the posttest within five days of the initial intake (during two separate sessions), due to scheduling constraints. Additionally, minor changes to the workshop at Location 2 were made to streamline the content, based on experience in conducting the workshop at the WJHTC. There were no changes to the interactive online module between administrations at the two locations. Therefore, results will be reported separately. All data were collected between July 2015 and September 2015. Population and Sampling After a thorough examination of accident reports, it was found the typical VFR-into-IMC accident pilot was a noninstrument rated pilot with less than 1000 hours of total time (Carney et al., 2015) . Therefore, researchers targeted this population for recruitment and utilized convenience sampling methods. Participants were contacted through a pilot database, word of mouth, and advertisement (flyers). Due to sample size goals, participants outside of the desired pilot profile were allowed to participate in the study. Location 1 had more variation and deviation from the desired profile than did Location 2. However, the distribution of participants at Location 1 was relatively balanced. Participants fitted the targeted profile more accurately at Location 2. Demographics and flight experience information for each location are reported in the Results section of this report. To account for variation, each participant was randomly assigned to one of the three experimental groups. Research Instrument To assess pilot knowledge and skills pertaining to VFR into-IMC, 24 weather-related questions were derived from the Private Pilot Airmen Certification Standards (ACS). According to the FAA (2015), ACS will replace existing practical test standards to provide enhanced and holistic testing for certificate applicants. The FAA targeted June 2016 for implementation of the new testing standards. The 24 questions were a combination of weather-related knowledge and skills questions. Questions 1–10 were knowledge questions, while questions 11–24 were skills questions. All questions were vetted through several rounds by seven aviation experts who had extensive flight and academic teaching experience, to increase external validity. Due to the close time proximity in which the participants took the pretest and posttest, the posttest consisted of slightly different questions from the same ACS categories. The posttest questions were presented in the same order as the pretest. Pretest and posttest questions are listed in Appendices A and B of this report. The private pilot weather-related ACS test questions codes can be viewed in Appendix C. A post-hoc internal consistency test was conducted for the pretest and posttest, with all scores combined respectively. The Cronbach’s alpha value for the pretest was 0.444 (total count 72) while the Cronbach’s alpha value for the posttest was 0.682 (total count 71). When comparing these values to a commonly accepted standard, the pretest had low internal consistency, while the posttest had an acceptable level of internal consistency. Interventions An interactive online training module and a workshop were created by the researchers. The topics presented for each intervention were similar; however, the workshop included accident case studies to foster discussion. Table 1 lists the interactive online course topics and Table 2 lists those for the workshop format. It was expected that one or both interventions would improve posttest scores. Participant demographic and flight experience data were collected as part of the experiment. This information included age, gender, total flight hours, accrued instrument time, and time flown in the previous six months. The demographic and flight experience data were sorted and depicted for each group, and these data are shown in Table 3. Table 4 shows participant flight experience information for each group, and Table 5 shows class of airplane most often flown and type of training received. Forty-eight participants started and completed the pretest and posttest (n 5 48) at Location 1. Once again, variation was caused by the need to include pilots outside of the desired profile to increase the sample size. This is further discussed in the limitations section. An ANCOVA, using SPSS 23H, was run to determine the effect of two different VFR-into-IMC training protocols and a control group on posttest scores after adjusting for pretest scores. There was an approximate linear relationship between pretest and posttest scores for each experimental group, as assessed by a visual inspection of a scatterplot. Homogeneity of regression slopes as the interaction term was not statistically significant, F(2,42) 5 1.486, p 5 0.238. Standardized residuals for the interventions and for the overall model were normally distributed, as assessed by the Shapiro–Wilk test (p . 0.05). There was homoscedasticity and homogeneity of variances, as assessed by visual inspection of a scatterplot and Levene’s test of homogeneity of variance (p 5 0.177), respectively. Though Barrlett’s test for variance is more sensitive, particularly for datasets that depart from normality, Levene’s test was used in both datasets due to the small sample size and normal distribution (National Institute of Standards and Technology, n.d.). No outliers were found in the data, as assessed by no cases with standardized residuals greater than ¡3 standard deviations. After adjustment for pretest scores, there was a statistically significant difference in posttest scores between the interventions, F(2,44) 5 4.923, p 5 0.012, gp2 5 0.183. Approximately 41% of the variance was explained by the pretest. The observed power for the ANCOVA test was 0.780. Power values above 0.80 are considered adequate. Interactive online group frequencies A post-hoc analysis was performed with a Bonferroni adjustment. The data provided sufficient evidence to indicate the workshop group posttest scores were significantly lower than those of the control group (p 5 0.009). There was no significant difference between the control group and the online group (p 5 0.340). Lastly, no significant difference was found between the online group and workshop group (p 5 0.411). Table 6 shows pretest statistics, and adjusted and unadjusted posttest values when using pretest scores as a covariate. Adjusted mean values were calculated from the multiple regression equation and are determined by how far above or below average the group was from the control variable (pretest). Figures 1–5 represent trends in the pretest and posttest scores. As seen in Figure 1, the pretest scores between the experimental groups were virtually the same. Questions 9 and 20 appear to have the lowest frequency of correct responses for all three groups. After the treatments (if applicable) were administered, the posttest line graph appears to be more sporadic. Line graphs were also created to show the comparison of pretest and posttest score for each experimental group separately. As with Location 1, demographics, flight experience, and category/class of aircraft most often flown are shown in Tables 7, 8, and 9, respectively. Twenty-three participants completed the posttests (n 5 23). One participant assigned to the online group elected to discontinue participation during the experiment. Therefore, all of the associated data for that participant were removed for the ANCOVA analysis. As with data from Location 1, an ANCOVA was run to determine the effect of two different VFR-into-IMC training protocols and a control group on posttest scores after adjusting for pretest scores for data collected at Location 2. There was an approximate linear relationship between pretest and posttest scores for each experimental group, as assessed by visual inspection of a scatterplot. Homogeneity of regression slopes as the interaction term was not statistically significant, F(2,17) 5 0.237, p 5 0.791. Standardized residuals for the interventions were normally distributed, as assessed by the Shapiro–Wilk test (p . 0.05). There was homoscedasticity and homogeneity of variances, as assessed by visual inspection of a scatterplot and Levene’s test of homogeneity of variance (p 5 0.112), respectively. There were no outliers in the data, as assessed by no cases with standardized residuals greater than ¡3 standard deviations. After adjustment for pretest scores, there was no statistical difference in posttest scores between the experimental groups, F(2,19) 5 0.696, p 5 0.511, gp2 5 0.068. Approximately 10% of the variance was explained by the pretest. The observed power for the ANCOVA test was 0.150. Power values above 0.80 are considered adequate. Table 10 depicts the pretest score statistics, unadjusted posttest scores, and adjusted posttest scores when using pretest scores as a covariate. Figures 6–10 represent trends in the pretest and posttest scores. As seen in Figure 6, the pretest scores between the experimental groups were similar. Pretest questions 9, 19, and 20 seemed problematic for the treatment groups. After the treatments, if applicable, were administered, the posttest line graph appears to have more variation. Posttest questions 9, 14, 16, and 24 had the lowest frequency of correct responses. Line graphs were also created to show the comparison of pretest and posttest scores for each experimental group separately. Discussion and Conclusions The purpose of this study was to develop training modules that would enhance GA pilots’ knowledge and skills pertaining to ACS weather-related questions. The treatments did not appear to significantly increase posttest scores when adjusted for pretest scores at either experiment location. Though previous research has shown the effectiveness of interactive online training modules and workshops (e.g., Silk, Perrault, Ladenson, & Nazione, 2015; Sitzmann et al., 2006) , enhancing GA pilot knowledge and skills in a short amount of time remains complex (Knecht et al., 2010) . Neither of the treatment groups yielded statistically higher mean posttest scores at either location. Moreover, the adjusted mean posttest score for the workshop group at Location 1 was significantly lower than that for the control group. This led the researchers to consider possible reasons for this result. As stated in the methodology, all participants completed not only the pretest and posttest, but also a flight training device session, which lasted approximately one hour. Therefore, the workshop group had the longest participation time and spent almost five hours at the facility. Additionally, there were reports that the flight training devices at Location 1 were difficult to control due to technical problems. Given these issues, it can be reasonable to assume fatigue may have played Note. SE 5 single engine; ME 5 multi-engine; CFI 5 certified flight instructor. a factor in the test results. It was also noted that the group average for pretest scores when combining both locations was 17.588 or 73%. This finding indicates that overall pilot weather knowledge may be considered low. The low overall average provides more evidence that weather-related aviation knowledge may be lacking among GA pilots and efforts should be continued to address the gaps. When reviewing the flight experience of participants, there were many that fell outside of the desired profile. There were commercial pilots, pilots holding an instrument rating, and pilots who may have had more than 1000 hours of total flight time among the test cohorts. In addition, the process researchers used to collect data may have been problematic. For example, when asking for total time, the highest option was 301+. It is unclear if participants in this category had 301 total hours or much more (e.g., 2000 total hours or 10,000 total hours). Random assignment may have accounted for the distribution of differences among participants. Lastly, when examining the mean posttest scores at Location 1, the difference between the control and workshop group was 2.674. Though this difference led to a statistically significant result, it may lack practical significance. The difference between two and three correct responses may be nominal. It is recommended that future researchers pay particular attention to the collection of demographic/flight experience information and participant fatigue concerns. Technical issues with flight training devices may occur unexpectedly, requiring difficult decisions to be made by researchers. Dividing the pretest, treatments (for those assigned), flight training device sessions, and posttest into two sessions at Location 2 helped to mitigate participant fatigue issues. In addition, there were no known issues with the flight training device used at Location 2. Additionally, the flight experience of the participants at Location 2 was much more uniform, and fitted the desired pilot profile more accurately. Only one participant indicated having more than 300 total flight hours. However, the results failed to reveal desirable findings. It is important to note that the two training modules utilized at each location were not designed to ‘‘coach’’ participants or teach to the test questions. It may be possible that immersive training could assist in enhancing knowledge and skills. Immersive training can include repetitive exposure to topics. For example, computer-based programs can assist in providing participants with visual cues, multiple weather reports, and decision-making opportunities. Utilizing different instructional methods or a combination of methods may provide deeper learning of the complex material. Exploring the length of time participants spend with researchers may also influence the results. It may be possible that a larger sample size could have yielded more desirable findings. Lastly, a pretest and posttest with higher internal consistency values may assist with measurement. Unfortunately, due to time constraints of the project, a pilot test was not conducted. It is suggested that future researchers conduct a beta test to establish acceptable reliability, since questions derived from the ACS do not guarantee high reliability. Furthermore, researchers may use this study to build upon the research design and understand potential limitations and successes. Although this study did not demonstrate significant differences between the control group and the treatment groups, VFR pilots should consistently address VFR-intoIMC concerns during preflight planning and in flight. While the GA community explores methods to mitigate VFR-into-IMC events, following recommended practices is essential. Pilots should self-study VFR-into-IMC material, which includes preflight planning, both preflight and inflight decision-making, operational pitfalls, the use of all available resources, and meteorological conditions that contribute to low visibility/low ceiling weather events. Preflight decision-making should include previous, current, and forecast weather reports. Go/no-go decisions must be made, based on the capability of the pilot and aircraft. Pilots should appropriately file flight plans and use inflight weather services. Recognition of deteriorating conditions should be based on reports and/or visual cues. Decisions must be made in a timely manner to avoid illegal or lessthan-desirable weather conditions. Conservative personal minima should be practiced and consistent evaluation of hazardous attitudes should be utilized. Proficiency in dealing with the aforementioned aspects should be discussed during pilot flight reviews. Lastly, regulators and organizations should increase promotion of resources, and pilots should take full advantage of those resources. Limitations Though this study did not yield the anticipated results, valuable knowledge was learned to assist researchers in this area. Future researchers can address the limitations that existed in this study. Limitations were found primarily in small sample sizes, low power found in the Location 2 data, demographic questions, issues with participant fatigue during the experiment (particularly for the workshop participants at Location 1), and technical issues with the simulators at Location 1. Future researchers can attempt to generate a larger sample size and consider two experimental groups, instead of three. This may help improve the power or the ability to detect statistical differences. The power found in the Location 1 dataset was 0.78 and is considered close to acceptable, while the power found in the Location 2 dataset was considerably lower at 0.15. As previously noted, specific questions in regards to demographics may leave researchers with less doubt regarding the extent of pilot experience. Additionally, restricting pilots outside of the desired profile from participating may increase demographic similarities, but there may be a tradeoff with sample size. Random assignment can account for differences and create a desired distribution among experimental groups. Researchers of the current study also acknowledge that participants may have faced fatigue issues, particularly with the workshop group at Location 1. Those participants spent approximately five hours in the research environment. Lastly, unforeseen technical issues may arise and disrupt the research process. This limitation, which occurred at Location 1, was challenging but was addressed appropriately by delaying the study until the issues were solved. Doing so, however, further decreased the sample size. Future researchers should have a sound plan for dealing with technical failures. Appendix A: Pretest Questions (PA.I.C.K4l) How will frost on the wings of an airplane affect takeoff performance? (PA.I.C.K4f) If an unstable air mass is forced upward, what type clouds can be expected? (PA.I.C.K4f) An almond or lens-shaped cloud which appears stationary, but which may contain winds of 50 knots or more, is referred to as (PA.I.C.K4e) One weather phenomenon which will always occur when flying across a front is a change in the (PA.I.C.K4g) Possible mountain wave turbulence could be anticipated when winds of 40 knots or greater blow (PA.I.C.K4j) One in-flight condition necessary for structural icing to form is (PA.I.C.K4f) The conditions necessary for the formation of cumulonimbus clouds are a lifting action and (PA.I.C.K4k) If the temperature/dew point spread is small and decreasing, and the temperature is 62˚F, what type weather is most likely to develop? (PA.I.C.K4k) Low-level turbulence can occur and icing can become hazardous in which type of fog? (PA.I.C.K4j) During an IFR cross-country flight you picked up rime icing which you estimate is 1/20 thick on the leading edge of the wings. You are now below the clouds at 2000 feet AGL and are approaching your destination airport under VFR. Visibility under the clouds is more than 10 miles, winds at the destination airport are 8 knots right down the runway, and the surface temperature is 3 degrees Celsius. You decide to (PA.I.C.S1) Which of the reporting stations have VFR weather? (Refer to Figure 12.) (PA.I.C.S1) What are the wind conditions at Wink, Texas (KINK)? (Refer to Figure 12.) (PA.I.C.S1) The base and tops of the overcast layer reported by a pilot are (refer to Figure 14) (PA.I.C.S1) If the terrain elevation is 1295 feet MSL, what is the height above ground level of the base of the ceiling? (Refer to Figure 14.) (PA.I.C.S2) From which primary source should information be obtained regarding expected weather at the estimated time of arrival if your destination has no Terminal Forecast? (PA.I.C.S5) Between 1000Z and 1200Z the visibility at KMEM is forecast to be (refer to Figure 15) (PA.I.C.S5) What is the forecast wind for KMEM from 1600Z until the end of the forecast? (Refer to Figure 15.) (PA.I.C.S1) What is the outlook for the southern half of Indiana after 0700Z? (Refer to Figure 16.) (PA.I.C.S1) The Chicago FA forecast section is valid until the twenty-fifth at (refer to Figure 16) (PA.I.C.S5) What is indicated when a current CONVECTIVE SIGMET forecasts thunderstorms? (PA.I.C.S4) Which in-flight advisory would contain information on severe icing not associated with thunderstorms? (PA.I.C.S1) What is the status of the front that extends from Nebraska through the upper peninsula of Michigan? (Refer to Figure 18.) (PA.I.C.S1) What weather phenomenon is causing IFR conditions in central Oklahoma? (Refer to Figure 18.) (PA.I.C.S1) What weather is forecast for the Florida area just ahead of the stationary front during the first 12 hours? (Refer to Figure 20.) Appendix B: Posttest Questions 1 (PA.I.C.K4l) Why is frost considered hazardous to flight? 2 (PA.I.C.K4f) What are characteristics of a moist, unstable air mass? 3 (PA.I.C.K4f) Crests of standing mountain waves may be marked by stationary, lens-shaped clouds known as 4 (PA.I.C.K4e) Steady precipitation preceding a front is an indication of 5 (PA.I.C.K4i) Where does wind shear occur? 6 (PA.I.C.K4j) In which environment is aircraft structural ice most likely to have the highest accumulation rate? 7 (PA.I.C.K4h) What conditions are necessary for the formation of thunderstorms? 8 (PA.I.C.K4k) In which situation is advection fog most likely to form? 9 (PA.I.C.K4k) Low-level turbulence can occur and icing can become hazardous in which type of fog? 10 (PA.I.C.K4j) During an IFR cross-country flight you picked up rime icing which you estimate is 1/20 thick on the leading edge of the wings. You are now below the clouds at 2000 feet AGL and are approaching your destination airport under VFR. Visibility under the clouds is more than 10 miles, winds at the destination airport are 8 knots right down the runway, and the surface temperature is 3 degrees Celsius. You decide to 11 (PA.I.C.S1) The wind direction and velocity at KJFK is from (refer to Figure 12) 12 (PA.I.C.S1) The remarks section for KMDW has RAB35 listed. This entry means (refer to Figure 12) 13 (PA.I.C.S1) The wind and temperature at 12,000 feet MSL as reported by a pilot are (refer to Figure 14) 14 (PA.I.C.S1) The intensity of the turbulence reported at a specific altitude is (refer to Figure 14) 15 (PA.I.C.S2) From which primary source should information be obtained regarding expected weather at the estimated time of arrival if your destination has no Terminal Forecast? 16 (PA.I.C.S5) In the TAF from KOKC, the ‘‘FM (FROM) Group’’ is forecast for the hours from 1600Z to 2200Z with the wind from (refer to Figure 15) 17 (PA.I.C.S5) In the TAF from KOKC, the clear sky becomes (refer to Figure 15) 18 (PA.I.C.S3, S1) What sky condition and visibility are forecast for upper Michigan in the eastern portions after 2300Z? (Refer to Figure 16.) 19 (PA.I.C.S1) What sky condition and type obstructions to vision are forecast for upper Michigan in the western portions from 0200Z until 0500Z? (Refer to Figure 16.) 20 (PA.I.C.S5) What information is contained in a CONVECTIVE SIGMET? 21 (PA.I.C.S4) Which in-flight advisory would contain information on severe icing not associated with thunderstorms? 22 (PA.I.C.S1) The IFR weather in northern Texas is due to (refer to Figure 18) 23 (PA.I.C.S1) According to the Weather Depiction Chart, the weather for a flight from southern Michigan to north Indiana is ceilings (refer to Figure 18) 24 (PA.I.C.S5) The enclosed shaded area associated with the low-pressure system over northern Utah is forecast to have (refer to Figure 20) Appendix C: Airmen Certification Standards—Weather Information Knowledge PA.I.C.K1 PA.I.C.K2 PA.I.C.K3 PA.I.C.K4 Aircraft Owners and Pilots Association . ( 2015 ). Accident Analysis . Retrieved from http://www.aopa.org/asf/ntsb/searchResults.cfm Ali , S. , Chalder , T. , & Madan , I. ( 2014 ). Evaluating interactive fatigue management workshops for occupational health professionals in the United Kingdom . Safety and Health at Work , 5 ( 4 ), 191 - 197 . Brooks-Harris , J. E. , & Stock-Ward , S. R. ( 1999 ). Workshops: Designing and facilitating experiential learning . Thousand Oaks , CA: Sage Publications. Carney , T. , Brown , L., Duncan , J. , Whitehurst , G. , Rantz , W. , Seiler , B. , … Mayes, P. ( 2015 ). Weather technology in the cockpit (WTIC) Project B: Unexpected transition from VFR to IMC. Unpublished WTIC B phase 1 final report . Coyne , J. T. , Baldwin , C. L. , & Latorella , K. A. ( 2008 ). Pilot weather assessment: Implications for visual flight rules flight into instrument meteorological conditions . International Journal of Aviation Psychology , 18 ( 2 ), 153 - 166 . Federal Aviation Administration. ( 2015 ). Fact sheet: General aviation safety . Retrieved from https://www.faa.gov/news/fact_sheets/news_ story. cfm?newsId521274 Guskey , T. R. ( 2014 ). Planning professional learning . Learning , 71 ( 8 ), 10 - 16 . Hunter , D. R. , Martinussen , M. , Wiggins , M. , & O'Hare , D. ( 2011 ). Situational and personal characteristics associated with adverse weather encounters by pilots . Accident Analysis & Prevention , 43 ( 1 ), 176 - 186 . Ison , D. ( 2014 ). Correlates of continued visual flight rules (VFR) into instrument meteorological conditions (IMC) general aviation accidents . Journal of Aviation/Aerospace Education & Research , 24 ( 1 ), 1 - 26 . Keller , J. C. ( 2015 ). Unexpected transition from VFR to IMC: An examination of training protocols to mitigate pilot gaps in knowledge and performance (unpublished doctoral dissertation) . Purdue University, West Lafayette, Indiana. Knecht , W. , Ball , J. , & Lenz , M. ( 2010 ). Effects of video weather training products, web-based preflight weather briefing, and local versus nonlocal pilots on general aviation pilot weather knowledge and flight behavior, phase I. Federal Aviation Administration , DOT/FAA/ AM-10/1, Office of Aerospace Medicine, Washington, DC. Retrieved from http://www.faa.gov/data_research/research/med_humanfacs/ oamtechreports/2010s/media/201001.pdf Knecht , W. R. , & Lenz , M. ( 2010 ). Causes of general aviation weatherrelated, non-fatal incidents: Analysis using NASA aviation safety reporting system data . Final Report DOT/FAA/AM-10/13. National Institute of Standards and Technology. (n.d.) . Levene's test for equality of variances . Retrieved from http://www.itl.nist.gov/div898/ handbook/eda/section3/eda35a.htm National Transportation Safety Board. ( 2012 ). NTSB identification: ERA12FA438 . Retrieved from http://www.ntsb.gov/_layouts/ntsb. aviation/brief2.aspx? ev_id520120711X12055&ntsbno5ERA12FA 438&akey51 Rust , C. ( 1998 ). The impact of educational development workshops on teachers' practice . International Journal for Academic Development , 3 ( 1 ), 72 - 80 . Silk , K. J. , Perrault , E. K. , Ladenson , S. , & Nazione , S. A. ( 2015 ). The effectiveness of online versus in-person library instruction on finding empirical communication research . The Journal of Academic Librarianship , 41 ( 2 ), 149 - 154 . Sitzmann , T. , Kraiger , K. , Stewart , D. , & Wisher , R. ( 2006 ). The comparative effectiveness of web-based and classroom instruction: A metaanalysis . Personnel Psychology , 59 ( 3 ), 623 - 664 . Wiggins , M. , & O'Hare , D. ( 2003 ). Weather-wise: Evaluation of a cuebased training approach for the recognition of deteriorating weather conditions during flight . Human Factors: The Journal of the Human Factors and Ergonomics Society , 45 ( 2 ), 337 - 345 . Wisher , R. A. , & Olson , T. M. ( 2003 ). The effectiveness of web-based training . Retrieved from http://indianstrategicknowledgeonline.com/ web/Effectiveness%20of% 20Web-based%20Training .pdf


This is a preview of a remote PDF: http://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1150&context=jate

Julius C Keller, Thomas Carney, Allen Xie, Wesley Major, Matt Price. VFR-into-IMC: An Analysis of Two Training Protocols on Weather-Related Posttest Scores, Journal of Aviation Technology and Engineering, 2017,