Randomized controlled comparison of cross-sectional survey approaches to optimize follow-up completeness in clinical studies
Randomized controlled comparison of cross- sectional survey approaches to optimize follow-up completeness in clinical studies
Regula S. von AllmenID 0 1
Christian TinnerID 1
J u?rg Schmidli 1
Hendrik T. Tevaearai 1
Florian Dick 0 1
0 Clinics for Vascular Surgery , Kantonsspital St. Gallen, St. Gallen , Switzerland , 2 Swiss Cardiovascular Centre, Department of Cardiovascular Surgery, University Hospital Bern and University of Bern , Bern , Switzerland
1 Editor: Iratxe Puebla, Public Library of Science , UNITED KINGDOM
In outcome research, incomplete follow-up is a major, yet potentially correctable source of bias. Cross-sectional surveys may theoretically increase completeness of follow-up, but low response rates are reported typically. We investigated whether a pre-notification letter improved patient availability for follow-up phone interviews and thereby improved cross-sectional survey yield.
Data Availability Statement: All relevant data are
within the paper and its Supporting Information
Funding: The author(s) received no specific
funding for this work.
Competing interests: The authors have declared
that no competing interests exist.
A consecutive series of vascular patients was randomly divided into a trial and a validation
population. The trial population was then randomized 1:1 to one of two cross-sectional
contact strategies: Strategy 1 consisted of direct contact attempts by up to 12 systematically
timed phone calls, whereas Strategy 2 used a personalized pre-notification letter to arrange
for scheduled phone call interviews. Response rates, average time and efforts needed per
patient and overall survey duration were compared. Subsequently, trial findings were
externally validated in the validation population.
Of 728 consecutive patients, 370 were allocated to the trial population. Trial patients
contacted by strategy 1 (n = 183) had a similar profile when compared to trial patients contacted
by strategy 2 (n = 187). Follow-up periods following surgery (54.3 versus 53.6 months) and
all-cause mortality rates (21.3% versus 18.7%) were comparable between the trial groups.
Cross-sectional information on survival outcomes was almost complete after both contact
strategies (99.5% versus 98.9%, P = 1.0). In 144/187 strategy 2 patients (77%) interviews
were scheduled successfully necessitating significantly less contact attempts (median of 1.3
versus 2.3 per patient, P<0.0001). However, invested time per patient was similar between
the groups (median of 10.1 versus 9.6 minutes), and survey strategy 1 completed earlier
(median time to contact 4 versus 11 days, P<0.0001). Therefore, strategy 1 was validated in
the validation population (n = 358): a low lost to follow-up rate below 1% (P = 1.0) was
reconfirmed necessitating an average of 2.3 contact attempts per patient.
Both contact strategies were equally successful in contacting almost all patients
cross-sectionally. If systematically timed, direct phone calls were less complicated to organize and
faster completed. Given the low time and effort per patient, outcome studies should invest in
systematic follow-up surveys to minimize attrition bias.
Validity of outcome research depends on completeness of follow-up information [
incomplete follow-up is associated with the risk of missing outcome events selectively.
Therefore, if two study groups differ in follow-up completeness, outcome comparisons may be
flawed by this particular kind of selection bias known as attrition bias. As a consequence,
follow-up information must be as complete as possible to minimise bias in outcome assessment
Risk of attrition bias is highest in observational studies, particularly if initiated posthoc and
if follow-up assessment is based on routine clinical aftercare. However, attrition bias may
theoretically also affect randomized controlled trials since the pre-interventional randomization
process can neither balance out nor preclude differences arising during outcome assessment.
Cross-sectional surveys are a possible method to obtain complete outcome information per
a given time point [
]. Depending on the study endpoint questionnaire surveys, phone
interviews and outpatient visits can theoretically be used. Questionnaires are least expensive but
carry a considerable risk of low response rates (i.e. selection bias) as well as information bias
]. In addition, they are confined to information that patients could be expected to self-report
in an accurate manner. Outpatient visits, on the other side of the spectrum, offer the advantage
that outcomes may be ascertained by trained individuals, but they are impractical for many
reasons including the impossibility to assess all patients simultaneously and that these
costintensive visits are rarely reimbursed if not driven clinically.
Standardized phone interviews have a number of advantages: they provide a direct patient
contact and are suitable for assessment of many endpoints. However, they are deemed time
consuming and may be frustrating if contact numbers are incorrect or patients cannot be
reached quickly. Even if successful, unexpected calls may come as a surprise to the patient
carrying the risk of recall and interviewer bias [
There is evidence that combining tracing methods may improve success of contact [
hypothesized that an advance notice might facilitate organization of telephone interviews in
terms of patient availability and disposition. Thus, this study assessed whether in
cross-sectional surveys a pre-notification letter to organize a phone interview improved efficacy of
telephone surveys while reducing typical disadvantages.
Materials and methods
This randomized controlled trial compared two distinct cross-sectional follow-up survey
strategies (contact strategy 1 versus contact strategy 2, see below) in randomly allocated patients.
2 / 13
Patients who had undergone open or endovascular abdominal aortic aneurysm (AAA) repair
between June 2001 and December 2010 at the University Hospital of Bern, Switzerland, were
eligible. They were identified from a prospective registry, and consecutive completeness was
cross-checked against all surgical records from the same time period [
]. Patients already
known to be dead were excluded, whereas assumed survivors were randomly divided 1:1 into
a trial and a validation population. The trial population was then divided again in a pragmatic
1:1 randomized, parallel-group study design with groups being allocated either to contact
strategy 1 or 2. Depending on trial outcome, the preferred strategy was to be validated within
the validation population.
All patients had formally consented in writing at the time of surgery to be contacted during
follow-up and for anonymized analyses (informed consent). This written consent to research
participation had been discussed and reconfirmed at every outpatient follow-up contact. No
posthoc changes were applied to the survey methods after the commencement of the study.
The present health research was deemed purely methodological and included no
therapeutic intervention in participants; therefore, the trial was not pre-registered. Study design and
analysis plan were approved by the cantonal research ethics committee Berne, Switzerland. All
data were anonymized before analysis.
Investigated contact strategies
The contact strategies were applied by the same investigator (CT): Contact strategy 1 consisted
of unannounced phone calls using the last registered phone number (Hospital administrative
database; Systems, Applications and Products, SAP, Walldorf, Germany). A maximum of six
contact attempts followed a predefined schedule (Fig 1). If attempts were not successful,
general practitioners and/or designated relatives were contacted to confirm if this patient was still
alive and whether the phone number had changed. Then the contact algorithm was restarted
using any updated contact details. Thus, the number of contact attempts was limited to a
theoretical maximum of twelve per patient. As last resort, municipal administrations were
inquired. Patients who could not be traced in any way were categorized ?lost to follow-up?.
Contact strategy 2, in contrast, sent personally signed letters to all participants first using
the last registered address in the hospital administrative database. These letters carried official
insignia including the phone number of the research office and explained the rationale behind
the present investigation. It also asked for reconfirmation of the registered phone number and
for a convenient time for the telephone interview within a small range of the predefined study
end date. An addressed and stamped envelope was attached for response. Responses were
registered and interviews scheduled at the indicated times. A latency of 16 days was accepted
before non-respondents were contacted according to the ?direct contact? strategy. Responders
who refused the interview were not contacted but included for survival analysis if the response
came from the patients themselves.
Structured interviews. Patient interviews were conducted in one of the three main Swiss
languages (i.e. German, French or Italian) and were all led by the same investigator (CT). A
standardised structure including the following 4 sets of standardized questions was used: (1)
were the patients able to recollect the aortic operation and type of repair (open versus
endovascular)? (2) how were present health state and physical fitness (using standardized categories,
see below)? (3) what was the degree of independency in daily living? And (4) had patients
undergone any intervention since the operation (either aneurysm-related or not)? (S1 and S2
Based on the patients? self-assessment, the subjective current overall health condition was
categorized according to a Likert scale ranging from ?excellent? over ?well? and ?fair? to
3 / 13
Fig 1. A scheme of the applied contract strategies. Fig 1 outlines the predefined contact schedule for the ?direct contact? (contact strategy 1) and the ?arranged
contact? (contact strategy 2) group.
?poor?. Physical fitness was measured by estimating maximum metabolic equivalents of task
(MET), which express the energy cost for specific physical activities in kcal/kg/hour. Typical
MET values range from 0.9 (sleeping) to 23 (running at 22.5 km/h pace). The compendium of
physical activities [
] was used for standardized MET assessment. Independency was
categorized into ?autonomous living?, ?needing support from relatives only?, ?depending on mobile
nursing services? or ?living in a nursing home?.
Baseline and outcome measures
Baseline patient characteristics included conventional demographics, date and type of initial
aortic operation and follow-up period until the study survey.
Primary outcome was the proportion of verified survival information at cross-sectional
survey (i.e., patient either alive or registered date of death versus patient being lost to follow-up).
Dates of death were ascertained either by information from patients? relatives or municipal
administrations. Kaplan-Meier curves were used to estimate cumulative survival rates based
on this information.
Secondary outcome measures included time and effort invested per survey-strategy (ie.,
number of phone calls, cumulative work time per patient, and time period between start of the
survey and the actual interview). Work time estimates per patient were given in minutes and
included (1) average time needed per interview (cumulative time spent for interviews divided
4 / 13
by the number of patients) and (2) the overall time invested per individual patient. The latter
summed up all contact attempts including those to relatives and general practitioners. For
contact strategy 2 an equal share of the 90 minutes needed to draft the invitation letter was added
to every patient plus an additional 3 min for dispatching the letters.
The ?elapsed time until interview? was measured in days and per patient: for contact strategy
1, it started for all patients at the day when contact attempts started and counted the days until
successful individual phone interviews. For contact strategy 2, in contrast, it started with the
dispatch of the invitation letters, which were all sent out at the same day.
All analyses were predefined, and the analysis plan was agreed before data were inspected.
Baseline information was retrieved electronically from the prospective patient registry.
Outcomes were collected on paper during the interviews. All data were subsequently anonymized
and transferred into a dedicated database (Microsoft Access. Redmond, Washington).
Analyses used IBM SPSS statistics (version 21.0 for Windows, IBM Corporation, Software Group,
Somers, New York). (S3 File)
To detect a minimum difference in follow-up completeness of 15% (72% versus 87% based
on previous study findings [
]) at a power of 90% and at a 5% alpha level, the required
minimum overall sample size was 300 (sample size calculator at clincalc.com) [
]. Adding a safety
margin of 20% (anticipated fraction of patients already known to be dead in the patient
population) the target size of the trial population was determined at 360.
Random allocation sequences (both for inclusion into either trial or validation population,
and within the trial population for assignment to one of two contact strategies, respectively)
were generated centrally using a freely available computer-based software (www.randomizer.
org). No restrictions were implemented to the type of randomization. The investigator
performing the survey (CT) could not be blinded for obvious reasons; neither could the patients
who however, were not aware of the trial before being contacted within the allocated strategy.
In contrast, the investigator performing the statistical analyses (RvA) was blinded towards
group allocation. The contact strategy with the superior outcome regarding follow-up
completeness and speed of survey completion was to be validated externally in the validation
Statistical methods. Endpoints were analyzed according to intention to treat. Assuming
an efficient randomization process, comparisons were not adjusted for confounding factors;
neither in the trial, nor during validation. Patient characteristics are described using
conventional summary statistics. Continuous variables were not assumed normally distributed and
therefore compared using Mann-Whitney-U test. Proportions were compared using Fisher?s
exact test. Median duration of the survey was estimated using Kaplan Meier curves, and time
to survey completion was compared using log rank-test. Validation results were compared
with the trial findings analogously. All tests were two-sided, and an alpha level of 0.05 was
chosen to assume statistical significance of differences.
The patient flow through the whole study including both, trial and subsequent validation, is
detailed in the CONSORT diagram (Fig 2). In brief, 766 patients were potentially eligible; 38
were excluded either because they were already known to be dead or because they were
duplicate entries in the database. Thus, 728 participants were included: 370 were randomly allocated
to the trial population and 358 to the validation population. Within the trial population, 183
participants were randomly assigned to the contact strategy 1, and 187 to the contact strategy
5 / 13
Fig 2. CONSORT diagram. The CONSORT diagram shows the patient flow through the study including trial and external validation.
2. No participant was excluded after group allocation. The whole trial population was
contacted between 21st February 2011 and 10th March 2011 using the allocated strategy, and all
were analyzed according to intention to treat. Secondary outcomes were obviously assessed
only in surviving participants who could be contacted. The validation process (n = 358) took
place between 14th March 2011 and 24th March 2011.
Baseline patient characteristics
(n = 183)
Fig 3. Time to successful interview. Fig 3 shows cumulative Kaplan Meier estimates of the proportion of successful interviews over time. Data are stratified for
the three groups: strategy 1 versus strategy 2 versus validation of strategy 1. The vertical dashed orange line marks the allowed pre-set latency of 16 days in the
?arranged contact? group (contact strategy 2) to await participants? responses. The ?direct contact? strategy (contact strategy 1) succeeded with a shorter median
duration to successful interview and was then re-evaluated in the validation group showing a similar duration to successful interview.
interview was scheduled (2 patients refused an interview and 21 response letters provided an
official date of death). The remaining 43 non-responders (23%) were contacted using the
?direct contact? strategy.
Number of phone calls. A total of 414 calls was needed for contact strategy 1 compared
with 246 for contact strategy 2. This corresponds to an average of 2.26 (IQR 1; 3) phone calls
per patient in contact strategy 1 versus 1.32 (IQR 1; 2, P<0.001) in contact strategy 2,
Invested work. In absolute terms, 30 hours and 44 minutes were invested to contact the
183 patients in the contact strategy 1 group, while 30 hours and 8 minutes were invested to
contact the 187 patients in the contact strategy 2 group, which included a 90 minutes surcharge
for drafting and 3 min per patient for dispatching the invitation letters. As a consequence, an
average of 10.1 versus 9.6 minutes was needed per patient.
8 / 13
The equally effective, but less complicated (no scheduled arrangements) and overall faster
strategy (i.e. strategy 1) was re-evaluated within the validation population (n = 358). Baseline
characteristics and overall mortality following the operation were similar as in both trial
groups (Table 1). Primary outcome did not differ either; the lost to follow-up rate was 0.6%
(n = 2; versus one in the trial contact strategy 1 group, P = 1.0) and an average of 2.34 call
attempts per patient was required which was also similar to the trial group (2.26 attempts per
patient, P = 0.795).
This study investigated and validated relatively inexpensive and readily available methods to
assess outcome information in cross-sectional surveys within clinical outcome research. The
main and somewhat surprising finding was not the absence of relevant differences between the
investigated contact strategies, but that successful and reliable tracing of more than 98% of
patients in relatively large study populations was reproducibly possible within a few days of a
predefined study end date and with an investigator investment of less than 15 minutes per
patient. Although the contact strategy relying on a pre-notification letter (contact strategy 2)
necessitated only half of contact attempts, it neither reduced overall investment of time and
effort nor did it improve follow-up completeness. In contrast, it increased procedural
complexity (sending the letters, monitoring of returns, scheduling the phone interviews according
to patient preferences and tracing non-responders) and delayed completion of the survey due
to the dispatch of the letter. Thus, the hypothesis of our study was not reconfirmed but the
overall goal of a complete cross-sectional follow-up was reached.
Completeness of follow up is critical because attrition bias can severely compromise
internal and external validity of study findings. It therefore needs to be measured and declared, for
instance by using the follow-up index [
]. As a rule of thumb it is considered that <5% loss of
patients during follow up leads to little bias, while >20% poses serious threats to validity, with
even less than 20% of incomplete follow up data being a problem [
]. The risk of bias however
is relative and depends on the investigated endpoint and population. For instance, missing
follow-up information regarding survival may affect outcome analyses more in frail patient
populations undergoing major surgery (e.g. patients undergoing AAA repair) than in a younger
and healthy patient cohort undergoing hernia repair.
Hard endpoints such as survival or subsequent operations usually represent the central
outcome measures in observational studies and can usually be assessed without inviting the
patient back to the outpatient clinic. Completeness of follow-up is best assured by one of three
ways. Prospective studies may implement a strictly scheduled follow-up surveillance (e.g. at 90
days, 1 and 3 years), which carries the advantage of equal follow-up periods for each patient
allowing for calculation of absolute outcome percentages at the predefined intervals. Although
this approach provides the most precise and reliable outcome estimates, it is work-intensive,
expensive and can rarely be implemented in clinical routine. In addition, it is only possible in
truly prospective research projects and restriction to identical follow-up periods in all patients
leads to loss of large proportions of potentially available follow-up information.
Alternatively patient samples may be cross-referenced with official and up to date registries
of death or interventions. This provides the most reliable, accessible and complete follow-up
information if all available patient information is to be used, but it provides only cumulative
outcome estimates (e.g. Kaplan Meier) and it is not available in most countries without a
national health service.
9 / 13
The last alternative is a comprehensive cross-sectional survey of the whole study population
at a predefined study end date, such as in the present study. Similarly to the use of official
registries this approach accepts variable follow-up periods for cumulative Kaplan Meier outcome
estimates, but researchers need to ensure up to date completeness of cross-sectional
]. This challenge is often avoided because of assumed impracticability and fear of failure,
but also because lack of follow-up completeness may easily pass unnoticed and the risk of
attrition bias is underestimated [
]. This study demonstrated that it is possible and worth taking
on the challenges of comprehensive cross-sectional surveys even in larger patient populations.
Regardless of the method, used return rates are crucial for survey validity particularly if the
reasons for (non)-responding may be causally linked to any of the collected outcome
information (e.g. deceased patients cannot respond) [
In postal questionnaire surveys, for instance, return rates usually range around 50 to 65%,
depending on patient demographics, provision of stamped return letters and bond between
patients and health care providers [
]. The same applies to typical contact rates in phone
Several approaches have been explored to increase survey completeness. In a Norwegian
study assessing 633 patients after surgery for degenerative disorders of the lumbar spine, some
78% responded to the initial postal questionnaire. Using systematic efforts to trace
nonresponders eventually resulted in a similar overall contact success as the present study, even if
not at the same speed. Only 2.8% had to be classified as lost to follow up [
Another approach to increase contact success consists of combining tracing methods, for
instance by notifying participants of a planned survey before the actual contact attempt.
However, scientific evidence of whether a pre-notification letter is beneficial in health research is
conflicting. In smaller scale studies and two small systematic reviews, a pre-notification letter
increased response rates considerably [
11, 12, 15
], whereas it had no impact on response rates
in a more recent larger scale study[
]. At least, respondents who received a pre-notification
letter were inclined to respond earlier [
Previous studies also assessed alternative strategies to increase response rates to mailed
questionnaires. For instance the odds of a response increased if a monetary incentive was used
], if the letter was signed individually and if a hand-addressed postage-paid return envelope
was added [
]. The present trial intervention (contact strategy 2) implemented several of
these approaches. The notification letter was signed individually containing a postage-paid
return envelope, and it was participant-friendly with easy-to-follow instructions asking only
very few questions, such as the preferred date and time for an interview and the current phone
number. Subsequently, it achieved an above-average early response rate of 77%. In the present
setting, however, contact strategy 1 yielded comparable contact rates.
This suggests that the study findings did not depend on particular aspects of any contact
strategy but were driven by the systematic overall approach. Both strategies used structured
contact sequences, starting by approaching patients via systematically timed phone calls,
contacting relatives and general practitioners and finally contacting municipal authorities. Such a
structured contact sequence is particularly important in an elderly population in which many
patients may have moved to a nursing home during the follow up time. In our present study,
some 3% of the patients lived in a nursing home at the time of the interview. Such highly
selected and probably most vulnerable patient subgroups should not be missed in
cross-sectional surveys to rule out selection bias. Assumedly, the preferred contact algorithm of a
crosssectional survey may be distinct in populations with different demographics.
Although the pre-notification letter failed to improve contact strategy 1, a potential benefit
lies in better informed patients at the time of the actual interview [
]. This advantage may be
accentuated if the interview is complex and asks for detailed information. Indeed, in the
10 / 13
present study, patients receiving a pre-notification recollected the type of aortic operation
significantly better than patients who were surprised by the direct phone call; however the
information advantage was not reflected in a shorter individual interview time or a higher degree of
accuracy of the remembered information.
A disadvantage of a notification letter is the potential prolongation of the overall survey
time, since dispatch of such letters and processing of the brief questionnaire may be associated
with a delay. Therefore, it is not surprising that contact strategy 1 completed the survey
significantly earlier, although it did not reduce the average time invested per patient. Also, the
efficiency of strategy 1 probably depends on the availability of the interviewer; in contrast, contact
strategy 2 allows the investigator efforts to be concentrated to scheduled interview slots.
Nowadays, many people are used to other forms of communication, such as short
messenger service (SMS), email or web-based electronic surveys. The effect of a SMS text notification
on the response rate to a postal questionnaire was investigated in a randomized controlled
trial. The authors found no impact of SMS on response rates, although there was a notion that
it might be effective in female participants [
]. Electronic surveys carry some obvious
advantages over other types of surveys; they are less costly, faster applied and more flexible than
postal or phone surveys, but they are also associated with significant disadvantages [
Generally, electronic surveys had lower response rates compared to more traditional survey
]. A successful completion of an electronic survey also relies on specific prerequisites,
such as availability of an accurate up-to-date and complete email address list for the
participants. This may be particularly relevant in an elderly group of participants, as in the present
study, since elderly people are less familiar with electronic communication. Many do still not
have an email address or computer and thus may not be contacted electronically [
The present study has particular strengths and limitations. The main strength of the study
lies in its experimental design. Due to randomization, the groups being investigated were
highly comparable with a low risk of confounding. The survey was conducted by a single
investigator according to a predefined contact algorithm, which reduced any bias associated
with interobserver variability. Moreover, a formal validation was performed in external
patients to reconfirm the trial findings. All of this suggests a high degree of internal and
external validity of study findings. The main study limitation may be seen in that it failed to link
any of the investigated interventions causally to cross-sectional survey success. This is most
likely due to the systematic structure of both approaches, but may also be influenced by the
investigated study population which consisted of elderly and typical vascular patients, living in
a middle European society where health service insurance is mandatory and the level of social
security is high. Therefore our findings may not be generalized to populations with different
ethnic, socioeconomic or demographic characteristics. For instance, it is possible that the same
trial in a younger (working) patient population would have led to different findings. In such a
setting, the preferred strategy might well have been to schedule interviews via a
pre-notification letter. However, the Norwegian study that assessed the behaviour to postal questionnaires
among patients operated for degenerative spine disorders has shown that particularly younger
patients are less likely to respond to postal surveys [
]. As a consequence, one may assume
that a systematically timed algorithm with various contact attempts at different daytimes may
compensate for demographic disparities among different patient populations and that such a
strategy may be applied for many types of postoperative follow up surveys.
Incomplete follow-up and low response rates to cross-sectional outcome assessments should
not be accepted as an intrinsic defect of outcome studies. Both investigated contact strategies
11 / 13
were equally successful in contacting almost all patients cross-sectionally. If systematically
timed, direct phone calls were less complicated to organize and completed faster. Given the
low time and effort per patient, outcome studies should invest in systematic cross-sectional
follow-up surveys to minimize risk of attrition bias.
S1 File. S1 File shows the standardized telephone survey questionnaire in English that was
used for the interview.
used for the interview.
S2 File. S2 File shows the standardized telephone survey questionnaire in German that was
S3 File. S3 File contains the collected baseline and outcome data.
Tevaearai, Florian Dick.
Conceptualization: Regula S. von Allmen, Ju?rg Schmidli, Hendrik T. Tevaearai, Florian Dick.
Data curation: Regula S. von Allmen, Christian Tinner.
Formal analysis: Regula S. von Allmen, Christian Tinner, Ju?rg Schmidli, Hendrik T.
Investigation: Regula S. von Allmen, Florian Dick.
Methodology: Regula S. von Allmen, Christian Tinner, Ju?rg Schmidli, Hendrik T. Tevaearai,
Project administration: Regula S. von Allmen.
Supervision: Regula S. von Allmen, Hendrik T. Tevaearai, Florian Dick.
Validation: Regula S. von Allmen, Christian Tinner, Florian Dick.
Visualization: Regula S. von Allmen, Florian Dick.
Writing ? original draft: Regula S. von Allmen, Christian Tinner, Ju?rg Schmidli, Hendrik T.
Writing ? review & editing: Regula S. von Allmen, Christian Tinner, Ju?rg Schmidli, Florian
Tevaearai, Florian Dick.
12 / 13
1. von Allmen RS , Weiss S , Tevaearai HT , Kuemmerli C , Tinner C , Carrel TP , et al. Completeness of Follow-Up Determines Validity of Study Findings: Results of a Prospective Repeated Measures Cohort Study . PLoS One . 2015 ; 10 ( 10 ):e0140817. https://doi.org/10.1371/journal.pone.0140817 PMID: 26469346; PubMed Central PMCID : PMCPMC4607456 .
2. Plos Medicine Editors. Better reporting of scientific studies: why it matters . PLoS Med . 2013 ; 10 ( 8 ): e1001504. https://doi.org/10.1371/journal.pmed.1001504 PMID: 24013839; PubMed Central PMCID : PMCPMC3754890 .
3. Clark TG , Altman DG , De Stavola BL. Quantification of the completeness of follow-up . Lancet . 2002 ; 359 ( 9314 ): 1309 - 10 . PMID: 11965278 .
4. Hazell ML , Morris JA , Linehan MF , Frank PI , Frank TL . Factors influencing the response to postal questionnaire surveys about respiratory symptoms . Prim Care Respir J . 2009 ; 18 ( 3 ): 165 - 70 . https://doi.org/ 10.3132/pcrj. 2009 .00001 PMID: 19104738 .
5. Choi BC , Pak AW . A catalog of biases in questionnaires . Prev Chronic Dis . 2005 ; 2(1):A13 . PMID: 15670466; PubMed Central PMCID : PMCPMC1323316 .
6. Booker CL , Harding S , Benzeval M. A systematic review of the effect of retention methods in population-based cohort studies . BMC Public Health . 2011 ; 11 : 249 . https://doi.org/10.1186/ 1471 -2458-11-249 PMID: 21504610; PubMed Central PMCID : PMCPMC3103452 .
7. Ainsworth BE , Haskell WL , Herrmann SD , Meckes N , Bassett DR Jr., Tudor-Locke C , et al. 2011 Compendium of Physical Activities: a second update of codes and MET values . Med Sci Sports Exerc . 2011 ; 43 ( 8 ): 1575 - 81 . https://doi.org/10.1249/MSS.0b013e31821ece12 PMID: 21681120 .
8. Siemiatycki J , Campbell S. Nonresponse bias and early versus all responders in mail and telephone surveys . Am J Epidemiol . 1984 ; 120 ( 2 ): 291 - 301 . PMID: 6465127 .
9. Rosner B . Fundamentals of Biostatistics. Boston te, editor: MA: Brooks/Cole; 2011 .
10. Dettori JR . Loss to follow-up . Evid Based Spine Care J . 2011 ; 2 ( 1 ): 7 - 10 . https://doi.org/10.1055/s0030-1267080 PMID: 22956930; PubMed Central PMCID : PMCPMC3427970 .
11. Edwards P , Roberts I , Clarke M , DiGuiseppi C , Pratap S , Wentz R , et al. Increasing response rates to postal questionnaires: systematic review . BMJ . 2002 ; 324 ( 7347 ): 1183 . PMID: 12016181; PubMed Central PMCID : PMCPMC111107 .
12. Nakash RA , Hutton JL , Jorstad-Stein EC , Gates S , Lamb SE . Maximising response to postal questionnaires-a systematic review of randomised trials in health research . BMC Med Res Methodol . 2006 ; 6:5 . https://doi.org/10.1186/ 1471 -2288-6-5 PMID: 16504090; PubMed Central PMCID : PMCPMC1421421 .
13. Smith W , Chey T , Jalaludin B , Salkeld G , Capon T. Increasing response rates in telephone surveys: a randomized trial . J Public Health Med . 1995 ; 17 ( 1 ): 33 - 8 . PMID: 7786565 .
14. Solberg TK , Sorlie A , Sjaavik K , Nygaard OP , Ingebrigtsen T. Would loss to follow-up bias the outcome evaluation of patients operated for degenerative disorders of the lumbar spine? Acta Orthop . 2011 ; 82 ( 1 ): 56 - 63 . https://doi.org/10.3109/17453674. 2010 .548024 PMID: 21189113; PubMed Central PMCID : PMCPMC3229998 .
15. Harrison RA , Cock D. Increasing response to a postal survey of sedentary patients-a randomised controlled trial [ISRCTN45665423] . BMC Health Serv Res . 2004 ; 4 ( 1 ): 31 . https://doi.org/10.1186/ 1472 - 6963-4-31 PMID: 15537429; PubMed Central PMCID : PMCPMC534787 .
16. Spruyt B , Van Droogenbroeck F. Forewarned is forearmed? A survey-experiment concerning the impact of pre-notification letters on response in a postal survey . Irish Journal of Sociology . 2014 ; 22 : 84 - 93 .
17. Brueton VC , Tierney JF , Stenning S , Meredith S , Harding S , Nazareth I , et al. Strategies to improve retention in randomised trials: a Cochrane systematic review and meta-analysis . BMJ Open . 2014 ; 4 ( 2 ):e003821. https://doi.org/10.1136/bmjopen-2013-003821 PMID: 24496696; PubMed Central PMCID : PMCPMC3918995 .
18. Scott P , Edwards P . Personally addressed hand-signed letters increase questionnaire response: a meta-analysis of randomised controlled trials . BMC Health Serv Res . 2006 ; 6 : 111 . Epub 2006/09/07. https://doi.org/10.1186/ 1472 -6963-6-111 PMID: 16953871; PubMed Central PMCID : PMC1574304 .
19. Sahlqvist S , Song Y , Bull F , Adams E , Preston J , Ogilvie D , et al. Effect of questionnaire length, personalisation and reminder type on response rate to a complex postal survey: randomised controlled trial . BMC Med Res Methodol . 2011 ; 11 : 62 . https://doi.org/10.1186/ 1471 -2288-11-62 PMID: 21548947; PubMed Central PMCID : PMCPMC3110121 .
20. Edwards PJ , Roberts I , Clarke MJ , Diguiseppi C , Wentz R , Kwan I , et al. Methods to increase response to postal and electronic questionnaires . Cochrane Database Syst Rev . 2009 ; (3):MR000008 . Epub 2009/07/10. https://doi.org/10.1002/14651858.MR000008.pub4 PMID: 19588449 .
21. Starr K , McPherson G , Forrest M , Cotton SC . SMS text pre-notification and delivery of reminder e-mails to increase response rates to postal questionnaires in the SUSPEND trial: a factorial design, randomised controlled trial . Trials . 2015 ; 16 : 295 . https://doi.org/10.1186/s13063-015-0808-9 PMID: 26152519; PubMed Central PMCID : PMCPMC4494692 .
22. Robson C , McCartan K. Real World Research: John Wiley & Sons Ltd; 2016 .
23. Scott A , Jeon SH , Joyce CM , Humphreys JS , Kalb G , Witt J , et al. A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors . BMC Med Res Methodol . 2011 ; 11 : 126 . https://doi.org/10.1186/ 1471 -2288-11-126 PMID: 21888678; PubMed Central PMCID : PMCPMC3231767 .
24. Anderson M , Perrin A . Tech adoptions climbs among older adults . Pew Research Center . 2017 .