2 Validating Tissue Signatures

The authors of BioQC have taken three independent approaches to show that their signatures are valid and biologically meaningful (Zhang et al. 2017). (1) They checked the results for batch effects using surrogate variable analysis, (2) they ensured that the signatures are biologically meaningful by relating them to biological knowledge, and (3) used an independent method to derive the signatures, which yields comparable results.

However, they do not quantify the performance of the signatures using standardized performance measures, such as sensitivity and specificity. In principle, one can think of three methods to achieve such a validation (Gönen 2009): (1) internal validation, i.e. using the same data for generating and testing signatures, (2) split-sample validation, i.e. dividing the dataset in a test and training dataset and (3) independent-sample validation, i.e. using entirely unrelated samples for training and testing. While method (2) might be acceptable if sufficient data is unavailable, only method (3) can ensure that the signature does not reflect experimental biases.

To address this, we independently derived signatures on the GTEx (Lonsdale et al. 2013) dataset using gini-index as described previously (Zhang et al. 2017), and performed both a 10-fold cross validation on the same dataset and a cross-species, cross-platform validation on the mouseGNF GeneAtlas (Lattin et al. 2008). To this end, we developed the python package pygenesig,¹ a framework to create and validate signatures.

In this chapter, we

perform a 10-fold cross-validation on the GTEx dataset, calculating the precision and recall for each signature,
perform a cross-species, cross-platform validation of the signatures generated on the GTEx dataset, and
identify a set of tissues, that can be reliably and unabmigously identified with the BioQC method.

2.1 Data

The Genotype Tissue Expression (GTEx) project (Lonsdale et al. 2013) is a comprehensive resource of tissue-specific gene expression data. We use this dataset to derive tissue-specific signatures. The data is human only and was generated using Illumina sequencing.
We use the GNF Mouse GeneAtlas V3 (Lattin et al. 2008) as a control dataset to demonstrate that the gini-method is robust over multiple platforms and species. This dataset originates from mouse and was generated using the Affymetrix Mouse Genome 430 2.0 Array (GPL1261).

2.2 Cross-Validation of signatures on the GTEx dataset

We use pygenesig to create and validate signatures on the GTEx v6 dataset. The data preparation steps are performed using these jupyter notebooks.² The output of pygenesig can be viewed here.³ Below, we summarize the analyses described in these documents.

We obtained the gene expression data and sample annotation from the GTEx portal. We collapsed gene expression data by HGNC symbol, aggregating by the sum. We aggregated samples of the same tissue by median. We generated signatures based on gini index ⁴ as described in the BioQC paper (Zhang et al. 2017). In brief, we calculated gini index for each gene across all tissues. Genes with a gini index \(\ge\) 0.8 and expression of \(\ge\) 5 TPM were added to the signatures of the 3 tissues with their highest expression.

Next, we performed a 10-fold cross-validation as follows: We split samples in 10 stratified folds,⁵ i.e. samples from all tissues are equally distributed across all folds. We use 9 folds to generate signatures. These signatures were applied to the remaining fold using BioQC. We iterated over the folds such that each fold has been used for training and testing.

Figure 2.1 shows the average BioQC score over all folds for each signature and each tissue.

cross-validation of GTEx tissue signatures. Signatures are shown on the y-axis, the corresponding groups of samples on the x-axis. The tile shading indicates the average BioQC score of a signature on a group of samples. For better visibility of low scores, the colors are saturated at 30.

Figure 2.1: cross-validation of GTEx tissue signatures. Signatures are shown on the y-axis, the corresponding groups of samples on the x-axis. The tile shading indicates the average BioQC score of a signature on a group of samples. For better visibility of low scores, the colors are saturated at 30.

As identifying contaminated or mislabled samples can be boiled down to a classification problem, we are interested in the predictive performance of each signature. Figure 2.2 shows the confusion matrix of using the signatures for classification. A sample is considered as classified as a tissue, if the corresponding signature scores highest among all other signatures.

Figure 2.2: Confusion matrix of the cross-validation.

2.3 Reference Signatures

From the above matrices we learn that, while the vast majority of signatures yield a high score in the corresponding tissue, an unambiguous classification of tissues is only viable for a subset of tissues. For instance, the different brain regions are hard to distinguish, and so are physiologically close tissues (e.g. large and small intestine).

Here, we reduce the dataset to a subset of tissues, which can be unambigously distinguished using the BioQC method (i.e. precision = recall = 1.0).

We manually map ⁶ the tissues from GTEx to a reduced subset of tissue names. The results are available in this jupyter notebook ⁷ and summarized below.

Figure 2.3 shows the confusion matrix of the reduced signature set. All tissues have been correctly identified at precision = recall = 1.0.

Figure 2.3: Confusion matrix of robust signatures.

2.4 Cross-Platform Cross-Species Validation

To asses if the signatures translate across species and platforms, we tested the signatures generated above (human, Illumina sequencing) on the mouseGNF tissue expression atlas (mouse, Affymetrix microarray). The procedure is described in this notebook ⁸ and summarized in this section.

Figure 2.4 shows the score matrix of GTEx signatures against mouseGNF samples.

Figure 2.4: Cross-platform, cross-species validation of the robust signatures identified in the previous step on mouse microarray data.

The signatures Brain, Heart, Kidney, Liver, Skeletal Muscle, Pancreas, Skin and Testis identify the respective tissue despite the species and platform differences at a high (>5) BioQC score.

As expected Heart and Skeletal muscle also identify each other, however Heart scores are still higher on heart samples and Skeletal muscle scores are higher on skeletal muscles samples, therefore we retain both signatures.

Surprisingly, Adrenal Gland, Ovary and Uterus are not able to identify the respective samples, despite having a high score in the cross-validation. We, therefore, exclude these signatures from the reference signature set.

Unfortunately, Blood was not profiled in the mouseGNF dataset. We keep the signature nonetheless as it does not trigger any false positives.

References

Gönen, Mithat. 2009. “Statistical aspects of gene signatures and molecular targets.” Gastrointest Cancer Res 3 (2 Suppl): 19–21. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2684735{\&}tool=pmcentrez{\&}rendertype=abstract.

Lattin, Jane E, Kate Schroder, Andrew I Su, John R Walker, Jie Zhang, Tim Wiltshire, Kaoru Saijo, et al. 2008. “Expression Analysis of g Protein-Coupled Receptors in Mouse Macrophages.” Immunome Research 4 (1): 5. https://doi.org/10.1186/1745-7580-4-5.

Lonsdale, John, Jeffrey Thomas, Mike Salvatore, Rebecca Phillips, Edmund Lo, Saboor Shad, Richard Hasz, et al. 2013. “The Genotype-Tissue Expression (GTEx) project.” Nature Genetics 45 (6): 580–85. https://doi.org/10.1038/ng.2653.

Zhang, Jitao David, Klas Hatje, Gregor Sturm, Clemens Broger, Martin Ebeling, Martine Burtin, Fabiola Terzi, Silvia Ines Pomposiello, and Laura Badi. 2017. “Detect tissue heterogeneity in gene expression data with BioQC.” BMC Genomics 18 (1): 277. https://doi.org/10.1186/s12864-017-3661-2.