RESPONSE: Re: Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK)

0 Journal of the National Cancer Institute , Vol. 97, No. 24, December 21, 2005 1 Oxford, U.K. (DGA); Institut fuer Medizinische Biometrie und Medizinische Informati, Universi- taetsklinikum Freiburg, Freiburg, Germany (WS); Centro Regionale Indicatori Biochimici di Tumore, Ospedale Civile , Venezia , Italy ( MG); OSI Phar- maceuticals, Inc., Boulder, CO (GMC). Biometric Research Branch, DCTD, National Can- cer Institute , Room 8126, Executive Plaza North, 6130 Executive Blvd., Bethesda, MD 20892-7434 ( 2 Affiliations of authors: Biometric Research Branch (LMM), Cancer Diagnosis Program (SET), DCTD, National Cancer Institute , Bethesda , MD; Centre for Statistics in Medicine , Wolfson College CORRESPONDENCE - We warmly welcome the National Cancer Institute-European Organisation for Research and Treatment of Cancer (NCI-EORTC) reporting guidelines for tumor marker prognostic studies (REMARK) recently reported in the Journal (1). We believe the authors are correct to point out the inadequacies in reporting results of many tumor markerprognostic studies and the difficulty in interpreting and comparing data from such articles (2). However, we feel that the authors have missed a major opportunity by falling short of mandating public access to raw time-to-event data. Although molecular markers that directly determine therapeutic efficacy play a major role in determining prognosis, other molecular determinants may only modulate patient outcome and are likely to have only a small to medium impact on overall patient survival. Most studies of prognostic molecular markers published to date have, however, been based on analyses of small sample sets that have inevitably been too underpowered to realistically determine the true relationship between a marker and patient prognosis. Pooling data from small studies by meta-analyses provides a means of generating more precise estimates of the true impact of markers without wasting considerable resources on clinical trials evaluating potentially nondiscriminating markers. We therefore applaud point 16 in the REMARK guidelines (mandatory citation of the multivariable effect ratio with appropriate confidence intervals), which will substantially aid meta-analysis of published literature by mandating suitable data points and will also help to avoid selection bias by reducing the number of excluded studies in which the marker effect could not be accurately reconstructed. Meta-analysis of individual patient data is, however, the gold standard for pooling time-to-event data (3), and its clinical utility in assessing therapeutic interventions has been proven. Unfortunately, this method of meta-analysis has had little impact in the field of molecular prognostics because of major constraints that include time, cost, and a requirement for collaboration (and is therefore prone to selection bias because of the potential for excluding datasets from noncollaborating groups) (4). Although others in the molecular marker community have recognized the potential biases associated with analysis of molecular data and have striven for improved clarity by means of public access (5), we feel that the REMARK guidelines have fallen short in this area and have left open the possibility of wasting precious scientific and public effort in analysis of futile markers. Specifically, we do not fully concur with the authors assertion that to do so would serve to propagate bad science. This claim is contrary to the fundamental principles of metaanalysis, a technique that, when correctly applied, has the ability not only to accurately gauge the true pooled effect but also to assess and to correct the causes of inconsistency (6). SANJAY POPAT RICHARD S. HOULSTON REFERENCES (1) McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, Clark GM, et al. Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK). J Natl Cancer Inst 2005;97:11804. (2) Simon R, Altman DG. Statistical aspects of prognostic factor studies in oncology. Br J Cancer 1994;69:97985. (3) Stewart LA, Parmar MK. Meta-analysis of the literature or of individual patient data: is there a difference? Lancet 1993;341:41822. (4) Piedbois P, Buyse M. Meta-analyses based on abstracted data: a step in the right direction, but only a first step. J Clin Oncol 2004;22: 383941. (5) Brazma A, Hingamp P, Quackenbush J, Sherlock G, Spellman P, Stoeckert C, et al. Minimum information about a microarray experiment (MIAME)-toward standards for microarray data. Nat Genet 2001;29:36571. (6) Egger M, Smith GD. Meta-analysis: potentials and promise. BMJ 1997;315:13714. NOTES We thank Popat and Houlston for their expression of support of the principles underpinning the REMARK guidelines, and we welcome further dialogue on the specific topic of public access to raw time-to-event data. Popat and Houlston suggest that we missed a major opportunity by falling short of mandating public access to raw time-to-event data. We mostly agree with Popat and Houlston on the idea of making raw data publicly accessible, as indicated by our statement we view movement in this direction as generally positive. However, we would like to explain our reasons for not bundling data access issues with the REMARK guidelines. We did not consider ourselves in a position to mandate anythingeither public access to raw data or adherence to the REMARK guidelines. Our approach with the REMARK guidelines was to make recommendations that were based on sound arguments and empirical evidence supporting the notion that adherence to the guidelines would benefit tumor marker research. An explanatory document in preparation will explicitly detail these arguments and evidence. Journal editors, funding agencies, and review bodies can mandate adherence, but they would wisely do so only if the rationale and benefits are clear. There are serious deficiencies in the quality of reporting of tumor marker studies. We maintain our position that more is better only when the data are of good quality. If studies are not clearly and fully reported, it may be impossible to distinguish badly designed or poorly executed studies from high-quality studies. Including data from poor studies in a meta-analysis might only add noise and obscure true findings. Even data from a high-quality study may be misinterpreted or misused if the details of the study design, execution, and analysis are not carefully documented. We believe that the problem of poor reporting of studies has to be tackled first, and we hope that the REMARK guidelines will lead to improvements. How to best implement public access to data must be carefully considered. If all journals were to require full public access to data from studies they published, this activity would capture much, but not all, of the useful data. The problem of publication bias is well recognized. For example, in an article by Kyzas et al. (1) published recently in this Journal, there were striking differences in estimated association be (...truncated)