Results 1  10
of
36
Correlation and LargeScale Simultaneous Significance Testing
 Journal of the American Statistical Association
"... Largescale hypothesis testing problems, with hundreds or thousands of test statistics “zi ” to consider at once, have become familiar in current practice. Applications of popular analysis methods such as false discovery rate techniques do not require independence of the zi’s, but their accuracy can ..."
Abstract

Cited by 97 (8 self)
 Add to MetaCart
Largescale hypothesis testing problems, with hundreds or thousands of test statistics “zi ” to consider at once, have become familiar in current practice. Applications of popular analysis methods such as false discovery rate techniques do not require independence of the zi’s, but their accuracy can be compromised in highcorrelation situations. This paper presents computational and theoretical methods for assessing the size and effect of correlation in largescale testing. A simple theory leads to the identification of a single omnibus measure of correlation. The theory relates to the correct choice of a null distribution for simultaneous significance testing, and its effect on inference. 1. Introduction Modern computing machinery and improved scientific equipment have combined to revolutionize experimentation in fields such as biology, medicine, genetics, and neuroscience. One effect on statistics has been to vastly magnify the scope of multiple hypothesis testing, now often involving thousands of cases considered simultaneously. The cases themselves are typically of familiar form, each perhaps a simple twosample comparison,
Largescale multiple testing under dependence
 J ROY STAT SOC B
, 2009
"... Summary. The paper considers the problem of multiple testing under dependence in a compound decision theoretic framework. The observed data are assumed to be generated from an underlying twostate hidden Markov model.We propose oracle and asymptotically optimal datadriven procedures that aim to mini ..."
Abstract

Cited by 25 (2 self)
 Add to MetaCart
Summary. The paper considers the problem of multiple testing under dependence in a compound decision theoretic framework. The observed data are assumed to be generated from an underlying twostate hidden Markov model.We propose oracle and asymptotically optimal datadriven procedures that aim to minimize the false nondiscovery rate FNR subject to a constraint on the false discovery rate FDR. It is shown that the performance of a multipletesting procedure can be substantially improved by adaptively exploiting the dependence structure among hypotheses, and hence conventional FDR procedures that ignore this structural information are inefficient. Both theoretical properties and numerical performances of the procedures proposed are investigated. It is shown that the procedures proposed control FDR at the desired level, enjoy certain optimality properties and are especially powerful in identifying clustered nonnull cases. The new procedure is applied to an influenzalike illness surveillance study for detecting the timing of epidemic periods.
Estimating false discovery proportion under arbitrary covariance dependence
 J. Amer. Statist. Assoc
, 2012
"... Multiple hypothesis testing is a fundamental problem in high dimensional inference, with wide applications in many scientific fields. In genomewide association studies, tens of thousands of tests are performed simultaneously to find if any SNPs are associated with some traits and those tests are c ..."
Abstract

Cited by 16 (4 self)
 Add to MetaCart
Multiple hypothesis testing is a fundamental problem in high dimensional inference, with wide applications in many scientific fields. In genomewide association studies, tens of thousands of tests are performed simultaneously to find if any SNPs are associated with some traits and those tests are correlated. When test statistics are correlated, false discovery control becomes very challenging under arbitrary dependence. In the current paper, we propose a novel method based on principal factor approximation, which successfully subtracts the common dependence and weakens significantly the correlation structure, to deal with an arbitrary dependence structure. We derive an approximate expression for false discovery proportion (FDP) in large scale multiple testing when a common threshold is used and provide a consistent estimate of realized FDP. This result has important applications in controlling FDR and FDP. Our estimate of realized FDP compares favorably with Efron (2007)’s approach, as demonstrated in the simulated examples. Our approach is further illustrated by
Inference with transposable data: modelling the effects of row and column correlations
 Journal of the Royal Statistical Society: Series B (Statistical Methodology
, 2012
"... Summary. We consider the problem of largescale inference on the row or column variables of data in the form of a matrix. Many of these data matrices are transposable meaning that neither the row variables nor the column variables can be considered independent instances. An example of this scenario ..."
Abstract

Cited by 13 (0 self)
 Add to MetaCart
(Show Context)
Summary. We consider the problem of largescale inference on the row or column variables of data in the form of a matrix. Many of these data matrices are transposable meaning that neither the row variables nor the column variables can be considered independent instances. An example of this scenario is detecting significant genes in microarrays when the samples may be dependent due to latent variables or unknown batch effects. By modeling this matrix data using the matrixvariate normal distribution, we study and quantify the effects of row and column correlations on procedures for largescale inference. We then propose a simple solution to the myriad of problems presented by unanticipated correlations: We simultaneously estimate row and column covariances and use these to sphere or decorrelate the noise in the underlying data before conducting inference. This procedure yields data with approximately independent rows and columns so that test statistics more closely follow null distributions and multiple testing procedures correctly control the desired error rates. Results on simulated models and real microarray data demonstrate major advantages of this approach: (1) increased statistical power, (2) less bias in estimating the false discovery rate, and (3) reduced variance of the false discovery rate estimators.
Are a set of microarrays independent of each other
, 2009
"... Having observed an m × n matrix X whose rows are possibly correlated, we wish to test the hypothesis that the columns are independent of each other. Our motivation comes from microarray studies, where the rows of X record expression levels for m different genes, often highly correlated, while the co ..."
Abstract

Cited by 12 (1 self)
 Add to MetaCart
(Show Context)
Having observed an m × n matrix X whose rows are possibly correlated, we wish to test the hypothesis that the columns are independent of each other. Our motivation comes from microarray studies, where the rows of X record expression levels for m different genes, often highly correlated, while the columns represent n individual microarrays, presumably obtained independently. The presumption of independence underlies all the familiar permutation, crossvalidation, and bootstrap methods for microarray analysis, so it is important to know when independence fails. We develop nonparametric and normaltheory testing methods. The row and column correlations of X interact with each other in a way that complicates test procedures, essentially by reducing the accuracy of the relevant estimators.
Hakonarson H: Multiple testing in genomewide association studies via hidden Markov models
 Bioinformatics
"... Motivation: Genome wide association studies (GWAS) interrogate common genetic variation across the entire human genome in an unbiased manner and hold promise in identifying genetic variants with moderate or weak effect sizes. However, conventional testing procedures, which are mostly pvalue based, ..."
Abstract

Cited by 9 (3 self)
 Add to MetaCart
(Show Context)
Motivation: Genome wide association studies (GWAS) interrogate common genetic variation across the entire human genome in an unbiased manner and hold promise in identifying genetic variants with moderate or weak effect sizes. However, conventional testing procedures, which are mostly pvalue based, ignore the dependency and therefore suffer from loss of efficiency. The goal of this article is to exploit the dependency information among adjacent SNPs to improve the screening efficiency in GWAS. Results: We propose to model the linear block dependency in the SNP data using hidden Markov Models. A compound decisiontheoretic framework for testing HMMdependent hypotheses is developed. We propose a powerful datadriven procedure (PLIS) that controls the false discovery rate (FDR) at the nominal level. PLIS is shown to be optimal in the sense that it has the smallest
Control of the false discovery rate under arbitrary covariance dependence
 J Am Stat Assoc
"... Multiple hypothesis testing is a fundamental problem in high dimensional inference, with wide applications in many scientific fields. In genomewide association studies, tens of thousands of tests are performed simultaneously to find if any genes are associated with some traits and those tests are ..."
Abstract

Cited by 7 (5 self)
 Add to MetaCart
Multiple hypothesis testing is a fundamental problem in high dimensional inference, with wide applications in many scientific fields. In genomewide association studies, tens of thousands of tests are performed simultaneously to find if any genes are associated with some traits and those tests are correlated. When test statistics are correlated, false discovery control becomes very challenging under arbitrary dependence. In the current paper, we propose a new methodology based on principal factor approximation, which successfully substracts the common dependence and weakens significantly the correlation structure, to deal with an arbitrary dependence structure. We derive the theoretical distribution for false discovery proportion (FDP) in large scale multiple testing when a common threshold is used and provide a consistent FDP. This result has important applications in controlling FDR and FDP. Our estimate of FDP compares favorably with Efron (2007)’s approach, as demonstrated by in the simulated examples. Our approach is further illustrated by some real data applications.
CHOOSING THE LESSER EVIL: TRADEOFF BETWEEN FALSE DISCOVERY RATE AND NONDISCOVERY RATE
"... Abstract: The problem of multiple comparisons has become increasingly important in light of the significant surge in volume of data available to statisticians. The seminal work of Benjamini and Hochberg (1995) on the control of the false discovery rate (FDR) has brought forth an alternative way of m ..."
Abstract

Cited by 4 (1 self)
 Add to MetaCart
Abstract: The problem of multiple comparisons has become increasingly important in light of the significant surge in volume of data available to statisticians. The seminal work of Benjamini and Hochberg (1995) on the control of the false discovery rate (FDR) has brought forth an alternative way of measuring type I error rate that is often more relevant than the one based on the familywise error rate. In this paper, we emphasize the importance of considering type II error rates in the context of multiple hypothesis testing. We propose a suitable quantity, the expected proportion of false negatives among the true alternative hypotheses, which we call nondiscovery rate (NDR). We argue that NDR is a natural extension of the type II error rate of single hypothesis to multiple comparisons. The utility of NDR is emphasized through the tradeoff between FDR and NDR, which is demonstrated using a few real and simulated examples. We also show analytically the equivalence between the FDRadjusted pvalue approach of Yekutieli and Benjamini (1999) and the qvalue method of Storey (2002). This equivalence dissolves the dilemma encountered by many practitioners of choosing the “right ” FDR controlling procedure. Key words and phrases: False discovery rate, genomescans, microarray data, multiple comparisons, multiple hypothesis testing, nondiscovery rate, power, type I error, type II error. 1.
Correcting Bias in Statistical Tests for Network Classifier Evaluation
, 2010
"... It is difficult to directly apply conventional significant tests to compare the performance of network classification models because network data instances are not independent and identically distributed. Recent work [12] has shown that paired ttests applied to overlapping network samples will resu ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
It is difficult to directly apply conventional significant tests to compare the performance of network classification models because network data instances are not independent and identically distributed. Recent work [12] has shown that paired ttests applied to overlapping network samples will result in unacceptably high levels (e.g., up to 50%) of Type I error (i.e., the tests lead to incorrect conclusions that models are different, when they are not). Thus, we need new strategies to accurately evaluate network classifiers. In this paper, we analyze the sources of bias (e.g. dependencies among network data instances) theoretically and propose analytical corrections to standard significant tests to reduce the Type I error rate to more acceptable levels, while maintaining reasonable levels of statistical power to detect true performance differences. We validate the effectiveness of the corrections empirically on both synthetic and real networks.
The joint null criterion for multiple hypothesis tests
 Stat. Appl. Genet. Mol. Biol
, 2011
"... Abstract Simultaneously performing many hypothesis tests is a problem commonly encountered in highdimensional biology. In this setting, a large set of pvalues is calculated from many related features measured simultaneously. Classical statistics provides a criterion for defining what a "corr ..."
Abstract

Cited by 4 (3 self)
 Add to MetaCart
(Show Context)
Abstract Simultaneously performing many hypothesis tests is a problem commonly encountered in highdimensional biology. In this setting, a large set of pvalues is calculated from many related features measured simultaneously. Classical statistics provides a criterion for defining what a "correct" pvalue is when performing a single hypothesis test. We show here that even when each pvalue is marginally correct under this single hypothesis criterion, it may be the case that the joint behavior of the entire set of pvalues is problematic. On the other hand, there are cases where each pvalue is marginally incorrect, yet the joint distribution of the set of pvalues is satisfactory. Here, we propose a criterion defining a well behaved set of simultaneously calculated pvalues that provides precise control of common error rates and we introduce diagnostic procedures for assessing whether the criterion is satisfied with simulations. Multiple testing pvalues that satisfy our new criterion avoid potentially large study specific errors, but also satisfy the usual assumptions for strong control of false discovery rates and familywise error rates. We utilize the new criterion and proposed diagnostics to investigate two common issues in highdimensional multiple testing for genomics: dependent multiple hypothesis tests and pooled versus testspecific null distributions.