Nuclear IHC enumeration: A digital phantom to evaluate the performance of automated algorithms in digital pathology

PLOS ONE, May 2018

Automatic and accurate detection of positive and negative nuclei from images of immunostained tissue biopsies is critical to the success of digital pathology. The evaluation of most nuclei detection algorithms relies on manually generated ground truth prepared by pathologists, which is unfortunately time-consuming and suffers from inter-pathologist variability. In this work, we developed a digital immunohistochemistry (IHC) phantom that can be used for evaluating computer algorithms for enumeration of IHC positive cells. Our phantom development consists of two main steps, 1) extraction of the individual as well as nuclei clumps of both positive and negative nuclei from real WSI images, and 2) systematic placement of the extracted nuclei clumps on an image canvas. The resulting images are visually similar to the original tissue images. We created a set of 42 images with different concentrations of positive and negative nuclei. These images were evaluated by four board certified pathologists in the task of estimating the ratio of positive to total number of nuclei. The resulting concordance correlation coefficients (CCC) between the pathologist and the true ratio range from 0.86 to 0.95 (point estimates). The same ratio was also computed by an automated computer algorithm, which yielded a CCC value of 0.99. Reading the phantom data with known ground truth, the human readers show substantial variability and lower average performance than the computer algorithm in terms of CCC. This shows the limitation of using a human reader panel to establish a reference standard for the evaluation of computer algorithms, thereby highlighting the usefulness of the phantom developed in this work. Using our phantom images, we further developed a function that can approximate the true ratio from the area of the positive and negative nuclei, hence avoiding the need to detect individual nuclei. The predicted ratios of 10 held-out images using the function (trained on 32 images) are within ±2.68% of the true ratio. Moreover, we also report the evaluation of a computerized image analysis method on the synthetic tissue dataset.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

http://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0196547&type=printable

Nuclear IHC enumeration: A digital phantom to evaluate the performance of automated algorithms in digital pathology

May Nuclear IHC enumeration: A digital phantom to evaluate the performance of automated algorithms in digital pathology Muhammad Khalid Khan Niazi 0 1 2 Fazly Salleh Abas 0 2 Caglar Senaras 0 1 2 Michael Pennell 0 2 Berkman Sahiner 0 2 Weijie Chen 0 2 John Opfer 0 2 Robert Hasserjian 0 2 Abner Louissaint 0 2 Arwa Shana'ah 0 2 Gerard Lozanski 0 2 Metin N. Gurcan 0 1 2 0 2 0 R01CA134451 (PIs: MG, GL), U24CA199374 (PIs: MG, AMadabhushi , AMartel), U01 CA198945 (PI: AB) from the National Cancer Institute. The work was also supported by OSU CCC Pelotonia Award (PI: Gurcan). The funders had no role in study 1 Center for Biomedical Informatics, Wake Forest School of Medicine , Winston-Salem , North Carolina, United States of America, 2 Faculty of Engineering and Technology, Multimedia University , Melaka , Malaysia , 3 Department of Biostatistics, College of Public Health, The Ohio State University , Columbus , Ohio, United States of America , 4 US Food and Drug Administration, Silver Spring, Maryland , United States of America, 5 Department of Psychology, The Ohio State University , Columbus , Ohio, United States of America, 6 Department of Pathology, Harvard Medical School , Boston , Massachusetts, United States of America, 7 Department of Pathology, The Ohio State University , Columbus, Ohio , United States of America 2 Editor: Kristen C. Maitland, Texas A&M University , UNITED STATES Automatic and accurate detection of positive and negative nuclei from images of immunostained tissue biopsies is critical to the success of digital pathology. The evaluation of most nuclei detection algorithms relies on manually generated ground truth prepared by pathologists, which is unfortunately time-consuming and suffers from inter-pathologist variability. In this work, we developed a digital immunohistochemistry (IHC) phantom that can be used for evaluating computer algorithms for enumeration of IHC positive cells. Our phantom development consists of two main steps, 1) extraction of the individual as well as nuclei clumps of both positive and negative nuclei from real WSI images, and 2) systematic placement of the extracted nuclei clumps on an image canvas. The resulting images are visually similar to the original tissue images. We created a set of 42 images with different concentrations of positive and negative nuclei. These images were evaluated by four board certified pathologists in the task of estimating the ratio of positive to total number of nuclei. The resulting concordance correlation coefficients (CCC) between the pathologist and the true ratio range from 0.86 to 0.95 (point estimates). The same ratio was also computed by an automated computer algorithm, which yielded a CCC value of 0.99. Reading the phantom data with known ground truth, the human readers show substantial variability and lower average performance than the computer algorithm in terms of CCC. This shows the limitation of using a human reader panel to establish a reference standard for the evaluation of computer algorithms, thereby highlighting the usefulness of the phantom developed in this work. Using our phantom images, we further developed a function that can approximate the true ratio from the area of the positive and negative nuclei, hence avoiding the need to detect individual nuclei. The predicted ratios of 10 held-out images using the function (trained on 32 images) are within ±2.68% of the true ratio. Moreover, we - design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors have declared that no competing interests exist. also report the evaluation of a computerized image analysis method on the synthetic tissue dataset. I. Introduction Immunohistochemical (IHC) staining of tissue sections is routinely used in pathology to aid in diagnosis and to characterize malignant tumors in humans and in animals [1±3]. It also plays a vital role in selecting an appropriate systemic therapy for cancer patients [ 2 ]. Certain stains are accepted in medical practice as important predictive markers of tumor aggressiveness (Ki67) or tumor type (ER, PR and HER2Neu) and are used to determine the type and intensity of therapy for a given patient [ 4 ]. All these markers are used according to specific guidelines where the intensity of stains and the number of positive cells are expressed as percentage of all malignant cells. Considering the importance of these markers, technical aspects of stains such as tissue fixation, tissue processing and quality of stains are becoming strictly regulated and monitored to assure reproducible results [ 5 ]. In the majority of cases, IHC stain interpretation is rendered by a trained pathologist using a light microscope and a manual method, which consists of counting each positively stained cell [ 5 ]. As expected, the manual method of enumeration of positive cells suffers from poor reproducibility even in the hands of an expert pathologist [ 6, 7 ]. Moreover, due to human reader limitations, true counting of IHC positive and negative cells can only be performed for a limited number of cells [8±11]. Thus, by necessity IHC stains are counted only in small fragments of tumors selected by the pathologist. Such random sampling may be representative of tumors with a homogenous pattern of staining; however, the majority of malignant tumors show a heterogeneous pattern of staining with some areas enriched in positive cells and others with only a few positive cells. Consequently, the counting of IHC positive cells using random sampling of small areas of IHC stained heterogeneous tumors is often biased and does not represent the true nature of the tumor [12]. Oversampling of predominantly positive areas results in overestimation, and oversampling of IHC poor areas results in underestimation of IHC positive cells. An alternative approach to quantify percentage of positive cells in a high number of cells is estimation based on visual perception of areas without actually counting the positive cells [ 6 ]. Due to the limited ability of the human reader to efficiently and reproducibly count hundreds or thousands of cells, the pathologist replaces counting with a more crude method such as estimation (gestalt) [ 9 ]. Not surprisingly, such a method introduces substantial bias and variability, and results in poor reproducibility. Considering these difficulties, it is logical that a marked effort is underway to develop computer algorithms to accurately, precisely and reproducibly enumerate IHC stained cells using high resolution images of histological sections. Using different approaches, several competing algorithms have been developed and are currently being tested for clinical use in pathology. Comparison of accuracy and reproducibility between these platforms is based on a reference standard established by human reader interpretation of IHC stains. Paradoxically, all computer method generated results are still compared to results that are generated by a human reader [ 13 ]. Using reference standards based on human reader interpretation of images hampers standardization of computer IHC methods mainly due to high inter- and intra-reader variability of humans. We have developed an in silico phantom comprised of both IHC positive and IHC negative (counterstained) cells with a known ground truth, thereby avoiding the need for a reference 2 / 14 standard built by human readers. This phantom is generated by our specialized software that uses large collections of high resolution images of different negative and positive cells and generates virtual tissue sections with a known percentage of positive and negative cells. The proportion can be varied from less than 1% to 100% of positive cells. The distribution of positive and negative cells can be adjusted to mimic true histological sections where positive cells can be distributed evenly throughout the tissue or may show focal clustering depending on the intention of the operator. A cluster of cells can be comprised of positive cells with different intensities of staining, different patterns of staining, different cell size with cells showing good separation and cells overlapping (to provide a more challenging task of cell segmentation for computer algorithms). II. Methods We extracted a group of Ki-67 positive and negative nuclei from images of Ki-67 stained lymphoma biopsies [ 6, 14 ]. Each group consists of single nuclei as well as nuclei clumps having two to five nuclei. For simplicity we call these nuclei (single or clumped) ªobjects.º In total, we have 50 Ki-67 positive objects and 40 Ki-67 negative objects. Fig 1 shows examples of some of these positive and negative objects. The IRB was approved (Study Number: 2007C0069) by the Office of Responsible Research Practices and The Institutional Review Board at the Ohio State University. It is worth mentioning that all images used in this study were fully anonymized. Let Li‡ and Lj denote the positive and negative objects, respectively. Here i = 1,2,3, 50 and j = 1,2,3, 40. To create a synthetic image IS, we perfrom the following steps. 1. Create canvas of 1000x1000 pixels. 2. For each pixel in the canvas, generate a three element vector (with elements corresponding to RGB colors) using a uniformly distributed pseudorandom integer generator between 190 and 250. These values serve as background for the canvas. 3. Select 900 equally spaced points/locations on the canvas. Let (xa,yb) represents the set of 900 locations, where a = 1,2,3, 30 and b = 1,2,3, 30. 4. Using a uniformly distributed pseudorandom integer generator, create a set of 900 pairs of random integers between -11 and +11. Let (Rxa,Ryb) represent the set of 900 value pairs. 5. Disturb the 900 equally spaced points by adding them with (Rxa,Ryb), i.e. (Pxa,Pyb) = (Rxa, Ryb) + (xa, yb). This will perturb the points slightly and will remove the systematic look of computer generated points. 6. To insert G% positive nuclei to the canvas, randomly draw positive objects with replacement from Li‡ until the total number of nuclei reaches Gx900/100. For instance, to create an image with 40% positive nuclei, we will keep on drawing positive objects until we have drawn a total of 360 nuclei. Then the selected objects are randomly scaled to resemble all different cell sizes in real Ki67 stained tissues. 7. Let us assume that we require N positive objects to create an image with G% positive nuclei. We place these N positive objects in randomly selected N locations from (Pxa,Pyb). 8. We repeat step 6 and 7 to place negative objects on the canvas. The only difference is that we exclude the N positions already filled by positive objects during extraction of random integers. As a final step, we perform Gaussian filtering on the edges of all positive and negative objects in the canvas. 3 / 14 Fig 1. Examples of positive and negative objects used in creation of synthetic images. All objects were cropped from Follicular Lymphoma whole slide images stained for Ki67. Using this method we created a set of 42 images with different concentrations of positive and negative nuclei. Fig 2 shows an example of two synthetic images created using this method. The 42 synthetic images created using the proposed method were used to investigate the following three questions on Estimation, Mapping, and Validation which are outlined next. II.a Estimation Is a visual estimation of the ratio of positive to total number of nuclei by a group of board certified pathologists consistent with the true ratio? Let rn denote true ratio (i.e., the ratio of positive to total number of nuclei during synthetic image generation) Fig 2. Examples of synthetic images generated by the proposed method. Left) An example image with only a few Ki67 positive nuclei. Right) An example image with higher concentration of Ki67 positive nuclei. 4 / 14 To answer the first question, a total of four board certified pathologists took part in the study. We shared the images through a customized web application, which allowed the pathologists to remotely view the images and provide their visual approximation of rn, which is denoted as Prjn (j = 1,..4). The pathologists also provided their visual approximation of the following ratio: rt ˆ Area of positive nuclei Area of the image II.b Validation Can one of our recent nuclei detection algorithms [ 15 ] approximate rn? In order to evaluate the robustness of the computer based solutions on the artificial dataset, we tested it against one of our recently published nuclei detection algorithms [ 15 ]. The algorithm starts by calculating two score maps for blue and brown objects, which correspond to negative and positive nuclei respectively. After calculating the score maps, the algorithm analyses the connected components for different threshold values and finds the optimal threshold value, which maximizes the number of separated nuclei. Finally, the algorithm uses this coarse segmentation result and splits the merged nuclei by detecting elliptical arcs and applying a super pixel method [ 16 ]. II.c Mapping Can we find a function C(ra) which maps ra ! rn, where ra corresponds to the ratio of the area of the positive nuclei to the total area of nuclei? The aim is to determine if we can replace the computationally expensive nuclei detection step with the less expensive area estimation. In order to answer the second question, we divided the 42 synthetic images into 10 subsets (SS1, SS2, SS3,. . .,SS10) as shown in Table 1. The third row in Table 1 contains the number of images in each subset. To approximate a function that maps ra ! rn, we randomly held out one image from each subset (a total of 10 images which we will use for testing) and computed the following difference for the remaining images (a total of 32 images which we used for training): III. Results For ease of comparison and to avoid redundancy, we combined the results of Subsections II.a (Estimation) and II.b (Validation) under Subsection III.a. The results for Subsection II.c are outlined in Subsection III.b. III.a Estimation & validation The results of our experiment are shown in Tables 2, 3 and 4. Table 2 shows the summary statistics of the pathologists and the computer algorithm in estimating rn (the ratio of true …1† …2† ra Mean ± SD Bias ± SD CCC (95% C.I.) Rank Corr. (95% C.I.) Pearson Corr. (95% C.I.) number of positive to total number of nuclei in an image). The four pathologists' estimation of rn is represented by Prin , where i 2 {1,2,3,4}, while the estimation by the computer algorithm is represented by Prcn . Likewise, Table 3 shows the summary statistics of the four pathologists and the computer algorithm in estimating rt (ratio of the area of the positive nuclei to the total area of the image). Here the total area of the image corresponds to the image size. Similarly, Table 4 6 / 14 Fig 3. Bland Altman Plots for rn. Bias and variability of the pathologists' estimates of rn increased with the percentage of positive nuclei. However, the bias and variability of the computer's estimate was relatively smaller than the pathologists' estimate of rn. shows the summary statistics of the four pathologists and the computer algorithm in estimating ra (the ratio of the true area of positive nuclei to area covered by all nuclei). It is worth mentioning that we never explicitly asked the pathologists to estimate ra. We used pathologists' estimates of rn as their surrogate estimate of ra in computing the performance metrics in Table 4 and our aim here was to check if the pathologists were unconsciously estimating ra when requested to estimate rn. CCC and C.I. in all three (Tables 2, 3 and 4) corresponds to Concordance Correlation Coefficient and Confidence Interval, respectively. Similarly Figs 3, 4 and 5 show the results in terms of Bland-Altman plots. These plots enable us to better understand bias and variability among pathologists and the computer algorithms relationship to ground truth. We also performed a separate analysis for each pathologist to allow the relationships to differ by reader. As expected, a pathologist's estimate of the percentage of positive nuclei can (almost) be perfectly explained by either the true percentage of positive cells rn and ra (see Table 5 for details). III.b Mapping Table 6 shows the approximation of rn for the 10 test images. Here rbn represents the approximation of rn through Eq 5. Table 6 was generated using leave-one-out cross validation methodology. Fig 6 shows a plot of D (as dots) for 32 training images along with a function that approximates these values. The horizontal axis in Fig 6 corresponds to ra while the vertical axis represents D. The objective is to find a function that maps ra to rn. The dotted lines in Fig 6 correspond to the 95% confidence interval. The continuous solid line represents a function 7 / 14 …3† …4† …5† …6† Fig 4. Bland Altman Plots for ra. The absolute bias and the variability of the pathologists' estimates of ra decreased with the increasing percentage of positive nuclei. This shows that for images with higher concentrations of positive nuclei, pathologists' estimates deviate from accuracy in ways that are not present in the algorithm's estimates. that approximates D using the following 2nd degree polynomial: D C…ra† ˆ a …ra†2 ‡ b ra ‡ c; where a ˆ 0:007; b ˆ 0:64; c ˆ 1:26 We determined the coefficients a,b,c by solving the following minimizing problem: a Here x corresponds to the vector of coefficients a,b,c, i.e., x ˆ 4 b 5. This means that rn can be approximated by substituting D with C(ra) in Eq 2, i.e. 2 3 c minxkD…x† r k a 2 C…ra† rn rn ra C…ra† IV. Discussion and conclusions The bias and agreement of the pathologists relative to the ground truth and those of our computer algorithm [ 15 ] are described in Tables 2±4 and the Bland-Altman Plots in Figs 3, 4 and 5. The level of agreement is captured in Tables 2±4 by the value of the CCC. Generally, the Fig 5. Bland Altman Plots for rt. There was a strong increasing linear relationship between pathologist bias in estimating rt and the percentage of positive nuclei. In contrast, there was an almost perfect inverse linear relationship between the bias of the computer's estimates and percentage of positive nuclei. larger values of CCC indicate better reproducibility between two variables. Pathologist bias and agreement with rn and ra were less biased and in better agreement than their estimates of rt. However, variability and absolute bias of the pathologists' estimates of rn increased as the number of positive cells increased (Fig 3) while the opposite was true when pathologist estimates were compared to ra (Fig 4). There was a strong increasing linear relationship between pathologist bias in estimating rt and the percentage of positive cells (Fig 5). The computer algorithm exhibited low bias and almost perfect agreement with rn and ra, but there was an almost perfect decreasing linear relationship between the computer algorithm bias and the percentage of positive nuclei. In contrast to the pathologist estimates, the variability of the computer's estimates was smaller and was independent of the number of positive nuclei. From the summarized statistics in Table 2, it is clear that visual estimations by pathologists are not a reliable means to approximate rn. Table 6 suggests that C(ra) provides a reasonable approximation of rn. The standard deviation of the differences between approximated values of rn and true rn is small (2.68%). State of rn 9 / 14 the art nuclei detection algorithms result in around 5±10% false detection rate [13, 17±19]. So, from that perspective, this is an extremely useful result as it 1) has the potential to replace computationally expensive nuclei detection algorithm with inexpensive area estimation, and 2) promises to produce more accurate or at least as good results as any state of the art nuclei detection method. But it still requires extensive validation and testing on a real dataset. At the moment, the synthetic nature of our images allowed us to compute ra by simply counting the number of positive and negative nuclei. However, the generalization of the mapping function to real data requires a selection of an automatic segmentation algorithm to approximate ra. Accurate IHC quantification is essential; improper quantification of IHC stains can deny a patient an important therapy option or result in unnecessary treatment. Unfortunately, IHC quantification can vary significantly between readers, as well as within readers [ 20, 21 ], for reasons that are neither fully understood nor easily addressed. Both quality assurance and pathologist training programs designed to standardize IHC stain interpretation suffer from lack of absolute standards with known ground-truth. Our Phantom generator allows creation of unlimited quantities and varieties of digital images that can be used for quality assurance programs, training of pathology residents and development of computer algorithms that will help to solve problem of poor reproducibility of IHC stains interpretation and quantification in clinical practice of pathology. IHC stains are used to characterize the repertoire of antigens (mostly proteins) that are expressed by malignant cells. Certain antigens are very specific and some are less specific but sensitive in determination of cell types and properties. For example most of carcinoma cells will express various keratin molecules and appear negative for common leukocyte antigen CD45. On the other hand, most leukemia/lymphoma cells will be negative for keratin and will be positive for CD45 antigen. Similarly different antigens are expressed by sarcoma cells. These antigens are negative in carcinoma and lymphoma cells while antigens expressed by carcinoma and lymphoma cells are negative in sarcoma cells. IHC stains are essential in detecting these differences and thus very useful in differentiating between carcinoma, lymphoma and sarcoma cells. Computationally, this finding suggests savings on computation time, considering that most nuclei detection algorithms [ 19, 22 ] take hours (if not days) to approximate the number of nuclei to facilitate the computation of rn. This mapping function if used in conjunction with the method in [ 6, 14 ] has the potential to reduce this time to minutes. However, this function (i.e.g C(A)) is specific to images of Ki-67 stained follicular lymphoma biopsies. Its generalization to other tumors may require re-estimation of C(A). It is important to perform a detailed analysis 10 / 14 Fig 6. The plot shows a function which facilitates the mapping of ra ! rn. The horizontal axis corresponds to different values of ra while the vertical axis represents D, i.e. error. Each individual dot represents the error between ra and rn for the training images. The solid line corresponds to the mapping function C which facilitates the mapping of ra to rn while the dotted lines represent the confidence interval. of Fig 6 to understand the quadratic behavior of the mapping function, C. When there are only a few positive nuclei, it results in the smallest error. This is evident from the y-axis of the plot in Fig 6; the errors are small when there are few positive nuclei and increase as the percentage of positive nuclei approaches 50%. The function starts to decrease in a quadratic fashion after this point until it reaches a point where the image only contains positive nuclei. So, it is essentially a function of homogeneity; it results in least amount of error when the image contains one type of nuclei. The function increases with the increase in heterogeneity. By generating a series of virtual tissues (phantoms) with different proportions and distribution of positive and negative cells, the exact number of positive and negative cells is known, therefore, it can be used as a gold standard for the following purposes: · An IHC standard for the industry to test computer algorithms for enumeration of IHC stained cells. · IHC standards for pathology quality assurance programs such as those administered by 9AP and similar organizations for programs used to standardize breast cancer pathology, lung cancer pathology, lymphoma pathology, etc. · Using these Phantom standards with different proportions of positive and negative cells, one can print hundreds of high resolution slides on glass that can be used for testing with light microscopy and for testing of high resolution slide scanners. · By further expanding this model, one can generate 3D phantoms of tissue with different proportions of negative and positive cells. Using 3D printers, one can print artificial tissue using a cartridge of collagen, or other matrix, with a positively and negatively stained suspension of cells. These 3D printed tissues could be used to standardize histology processing of tissue fixation and tissue cutting. Using 3D printers with cartridges of collagen or other Matrix and unstained cells with a known immunophenotype, one can generate (print) 3D artificial tissues that can be used as a standard for tissue fixation, tissue processing, IHC staining using different IHC platforms, image acquisition, and image IHC analysis. The main objective of this study was to provide a reliable approach to generate ground truth in the study and analysis of IHC stained tissues. In most studies of IHC stained images, 11 / 14 obtaining the ground truth is a major challenge due to various factors such as limited availability of experts to perform annotations, inter- and intra reader-variability and limited datasets. Often, the ground truth is either a generalization of reader consensus or the average of readings among multiple readers which may be biased, hence disputed. This method allows generating virtual images mimicking IHC stained tissue with exact proportions of positive and negative nuclei with controlled distribution of stained nuclei. These virtual tissue images can be used as the gold standard for IHC quantification for both computer based image analysis methods and quality assurance programs for manual evaluation of IHC stained tissue by pathologists. It creates virtual IHC stained sections with a known percentage of positive and negative cells and eliminates bias that is associated with IHC standards where the ground truth is based on manual counting of positive nuclei by pathologists. The study also showed that it is possible to approximate the ratio of positive to total number of nuclei from areas of positive and negative nuclei. The study also evaluated the performance of one of our recently developed nuclei detection algorithms [ 15 ]. In the future, we are planning on generating whole slide images which contain structures like blood vessels and other types of nuclei. We are also planning on extending it to other disease types including lymphoma, different types of breast cancer, lung cancer, prostate cancer, etc. For each tumor type phantom tissue can be further complicated by introduction of other objects such as lymphocytes, histiocytes, blood vessels, nerve bundles, fibrotic fibers, and artifacts such as hemorrhage of necrosis. The current approach lacks the ability to generate complex structures like ducts and lobules in breast. Moreover, it is relatively hard to generate a particular distribution of nuclei with the current design. Similarly, it requires a bit of tuning if the objective is to orient nuclei in a certain direction. Supporting information S1 File. Interactive Image Compression for Big Data Image Analysis: Application to Hotspot Detection in breast cancer. (PDF) Acknowledgments The project described was supported in part by Awards Number R01CA134451 (PIs: Gurcan, Lozanski), U24CA199374 (PIs: Gurcan, Madabushi, Martel), and U01 CA198945 (PI: Bilgin) from the National Cancer Institute, and OSU CCC Pelotonia Award. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute, or the National Institutes of Health. Author Contributions Conceptualization: Metin N. Gurcan. Data curation: Muhammad Khalid Khan Niazi, Robert Hasserjian, Abner Louissaint, Jr., Arwa Shana'ah, Gerard Lozanski. Formal analysis: Muhammad Khalid Khan Niazi, Michael Pennell, Berkman Sahiner, Weijie Chen, Metin N. Gurcan. Funding acquisition: Metin N. Gurcan. Methodology: Metin N. Gurcan. 12 / 14 Project administration: Metin N. Gurcan. Software: Muhammad Khalid Khan Niazi, Fazly Salleh Abas, Caglar Senaras. Validation: Muhammad Khalid Khan Niazi, Caglar Senaras, Michael Pennell, Berkman Sahiner, Weijie Chen, Robert Hasserjian, Abner Louissaint, Jr., Arwa Shana'ah, Metin N. Gurcan. Visualization: Muhammad Khalid Khan Niazi. Writing ± original draft: Muhammad Khalid Khan Niazi, Michael Pennell, Metin N. Gurcan. Writing ± review & editing: Muhammad Khalid Khan Niazi, Fazly Salleh Abas, Caglar Senaras, Michael Pennell, Berkman Sahiner, Weijie Chen, John Opfer, Robert Hasserjian, Abner Louissaint, Jr., Arwa Shana'ah, Gerard Lozanski, Metin N. Gurcan. 13 / 14 1. Zaha DC . Significance of immunohistochemistry in breast cancer . World journal of clinical oncology . 2014 ; 5 ( 3 ): 382 . https://doi.org/10.5306/wjco. v5.i3.382 PMID: 25114853 2. Niazi MKK , Downs-Kelly E , Gurcan MN , editors. Hot spot detection for breast cancer in Ki-67 stained slides: image dependent filtering approach . SPIE Medical Imaging; 2014 2014: International Society for Optics and Photonics . 3. Di Cataldo S , Ficarra E , Acquaviva A , Macii E. Automated segmentation of tissue images for computerized IHC analysis . Computer methods and programs in biomedicine. 2010 ; 100 ( 1 ):1± 15 . https://doi.org/ 10.1016/j.cmpb. 2010 . 02 .002 PMID: 20359767 4. de Matos LL , Trufelli DC , de Matos GL , da Silva Pinhal A. Immunohistochemistry as an important tool in biomarkers detection and clinical practice . Biomarker insights . 2010 ; 5:9 . PMID: 20212918 5. Taylor C , Levenson RM . Quantification of immunohistochemistryÐissues concerning methods, utility and semiquantitative assessment II . Histopathology. 2006 ; 49 ( 4 ): 411 ± 24 . https://doi.org/10.1111/j. 1365- 2559 . 2006 . 02513 . x PMID : 16978205 6. Niazi MKK , Pennell M , Elkins C , Hemminger J , Jin M , Kirby S , et al., editors. Entropy based quantification of Ki-67 positive cell images and its evaluation by a reader study . SPIE Medical Imaging; 2013: International Society for Optics and Photonics. 7. Niazi MKK , Downs-Kelly E , Gurcan MN , editors. Hot spot detection for breast cancer in Ki-67 stained slides: image dependent filtering approach . SPIE Medical Imaging; 2014: International Society for Optics and Photonics. 8. Niazi MKK , Satoskar AA , Gurcan MN , editors. An automated method for counting cytotoxic T-cells from CD8 stained images of renal biopsies . SPIE Medical Imaging; 2013: International Society for Optics and Photonics. 9. Revkin SK , Piazza M , Izard V , Cohen L , Dehaene S. Does subitizing reflect numerical estimation? Psychological Science . 2008 ; 19 ( 6 ): 607 ± 14 . https://doi.org/10.1111/j.1467- 9280 . 2008 . 02130 . x PMID : 18578852 10. Mandler G , Shebo BJ . Subitizing: An analysis of its component processes . Journal of Experimental Psychology: General , 111 , 1± 22 . Journal of Experimental Psychology: General. 1982 ; 111 ( 1 ). 11. Frith CD , Frit U. The solitaire illusion: An illusion of numerosity . Attention, Perception, & Psychophysics . 1972 ; 11 ( 6 ): 409 ± 10 . 12. Belhomme P , Toralba S , Plancoulaine B , Oger M , Gurcan MN , Bor-Angelier C . Heterogeneity assessment of histological tissue sections in whole slide images . Computerized Medical Imaging and Graphics . 2015 ; 42 : 51 ±5. https://doi.org/10.1016/j.compmedimag. 2014 . 11 .006 PMID: 25475487 13. Xu H , Lu C , Berendt R , Jha N , Mandal M. Automatic nuclei detection based on generalized laplacian of gaussian filters . IEEE Journal of Biomedical and Health Informatics . 2016 ; 21 ( 3 ): 826 ± 37 . https://doi. org/10.1109/JBHI. 2016 .2544245 PMID: 28113876 14. Niazi MKK , Lin Y , Liu F , Ashoka A , Marcellin M , Tozbikian G , et al. Interactive Image Compression for Big Data Image Analysis: Application to Hotspot Detection in Breast Cancer. Submitted for Journal publication . 2017 . 15. senaras; C, Pennell; M, Chen ; W, Sahiner; B, Shana'ah; A , Louissaint ; A , et al., editors. FOXP3 -stained image analysis for follicular lymphoma: Optimal adaptive thresholding with maximal nucleus coverage . SPIE Medical Imaging 2017 ; Orlando, USA. 16. Pătrăucean V , Gurdjos P , Von Gioi RG. A parameterless line segment and elliptical arc detector with enhanced ellipse fitting . Computer Vision ±ECCV Springer; 2012 . p. 572 ± 85 . 17. Chen T , Chefd'hotel C, editors. Deep learning based automatic immune cell detection for immunohistochemistry images . International Workshop on Machine Learning in Medical Imaging; 2014 : Springer. 18. Mao Y , Yin Z , Schober JM , editors. Iteratively training classifiers for circulating tumor cell detection . IEEE 12th International Symposium on Biomedical Imaging (ISBI) 2015 : IEEE. 19. Gurcan MN , Boucheron LE , Can A , Madabhushi A , Rajpoot NM , Yener B . Histopathological image analysis: A review . IEEE reviews in biomedical engineering . 2009 ; 2 : 147 ± 71 . https://doi.org/10.1109/ RBME. 2009 .2034865 PMID: 20671804 20. Hermans J , Krol A , Van Groningen K , Kluin PM , Kluin-Nelemans J , Kramer M , et al. International Prognostic Index for aggressive non-Hodgkin's lymphoma is valid for all malignancy grades . Blood . 1995 ; 86 ( 4 ): 1460 ± 3 . PMID: 7632953 21. Metter GE , Nathwani BN , Burke JS , Winberg CD , Mann RB , Barcos M , et al. Morphological subclassification of follicular lymphoma: variability of diagnoses among hematopathologists, a collaborative study between the Repository Center and Pathology Panel for Lymphoma Clinical Studies . Journal of Clinical Oncology . 1985 ; 3 ( 1 ): 25 ± 38 . https://doi.org/10.1200/JCO. 1985 . 3 .1.25 PMID: 3965631 22. Sirinukunwattana K , Raza SEA , Tsang Y-W , Snead DR , Cree IA , Rajpoot NM . Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images . IEEE transactions on medical imaging . 2016 ; 35 ( 5 ): 1196 ± 206 . https://doi.org/10.1109/TMI. 2016 .2525803 PMID: 26863654


This is a preview of a remote PDF: http://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0196547&type=printable

Muhammad Khalid Khan Niazi, Fazly Salleh Abas, Caglar Senaras, Michael Pennell, Berkman Sahiner, Weijie Chen, John Opfer, Robert Hasserjian, Abner Louissaint, Arwa Shana'ah, Gerard Lozanski, Metin N. Gurcan. Nuclear IHC enumeration: A digital phantom to evaluate the performance of automated algorithms in digital pathology, PLOS ONE, 2018, DOI: 10.1371/journal.pone.0196547