Results 1  10
of
37
Formalized data snooping based on generalized error rates. Econometric Theory
, 2008
"... It is common in econometric applications that several hypothesis tests are carried out simultaneously+ The problem then becomes how to decide which hypotheses to reject, accounting for the multitude of tests+ The classical approach is to control the familywise error rate ~FWE!, which is the probabil ..."
Abstract

Cited by 33 (9 self)
 Add to MetaCart
It is common in econometric applications that several hypothesis tests are carried out simultaneously+ The problem then becomes how to decide which hypotheses to reject, accounting for the multitude of tests+ The classical approach is to control the familywise error rate ~FWE!, which is the probability of one or more false rejections+ But when the number of hypotheses under consideration is large, control of the FWE can become too demanding+ As a result, the number of false hypotheses rejected may be small or even zero+ This suggests replacing control of the FWE by a more liberal measure+ To this end, we review a number of recent proposals from the statistical literature+ We briefly discuss how these procedures apply to the general problem of model selection+ A simulation study and two empirical applications illustrate the methods+ 1.
Stepup procedures for control of generalizations of the familywise error rate
 Ann. Statist
, 2006
"... Consider the multiple testing problem of testing null hypotheses H1,...,Hs. A classical approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate (FWER), the probability of even one false rejection. But if s is large, control of t ..."
Abstract

Cited by 31 (9 self)
 Add to MetaCart
(Show Context)
Consider the multiple testing problem of testing null hypotheses H1,...,Hs. A classical approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate (FWER), the probability of even one false rejection. But if s is large, control of the FWER is so stringent that the ability of a procedure that controls the FWER to detect false null hypotheses is limited. It is therefore desirable to consider other measures of error control. This article considers two generalizations of the FWER. The first is the kFWER, in which one is willing to tolerate k or more false rejections for some fixed k ≥ 1. The second is based on the false discovery proportion (FDP), defined to be the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289–300] proposed control of the false discovery rate (FDR), by which they meant that, for fixed α, E(FDP) ≤ α. Here, we consider control of the FDP in the sense that, for fixed γ and α, P {FDP> γ} ≤ α. Beginning with any nondecreasing sequence of constants and pvalues for the individual tests, we derive stepup procedures that control each of these two measures of error control without imposing any assumptions on the dependence structure of the pvalues. We use our results to point out a few interesting connections with some closely related stepdown procedures. We then compare and contrast two FDPcontrolling procedures obtained using our results with the stepup procedure for control of the FDR of Benjamini and Yekutieli [Ann. Statist. 29 (2001) 1165–1188]. 1. Introduction. In
Multiple testing and error control in Gaussian graphical model selection
 Statistical Science
"... Abstract. Graphical models provide a framework for exploration of multivariate dependence patterns. The connection between graph and statistical model is made by identifying the vertices of the graph with the observed variables and translating the pattern of edges in the graph into a pattern of cond ..."
Abstract

Cited by 28 (4 self)
 Add to MetaCart
Abstract. Graphical models provide a framework for exploration of multivariate dependence patterns. The connection between graph and statistical model is made by identifying the vertices of the graph with the observed variables and translating the pattern of edges in the graph into a pattern of conditional independences that is imposed on the variables ’ joint distribution. Focusing on Gaussian models, we review classical graphical models. For these models the defining conditional independences are equivalent to vanishing of certain (partial) correlation coefficients associated with individual edges that are absent from the graph. Hence, Gaussian graphical model selection can be performed by multiple testing of hypotheses about vanishing (partial) correlation coefficients. We show and exemplify how this approach allows one to perform model selection while controlling error rates for incorrect edge inclusion. Key words and phrases: Acyclic directed graph, Bayesian network, bidirected graph, chain graph, concentration graph, covariance graph, DAG, graphical model, multiple testing, undirected graph. 1.
SOME NONASYMPTOTIC RESULTS ON RESAMPLING IN HIGH DIMENSION, I: CONFIDENCE REGIONS
 SUBMITTED TO THE ANNALS OF STATISTICS
, 2009
"... We study generalized bootstrap confidence regions for the mean of a random vector whose coordinates have an unknown dependency structure. The random vector is supposed to be either Gaussian or to have a symmetric and bounded distribution. The dimensionality ..."
Abstract

Cited by 17 (1 self)
 Add to MetaCart
(Show Context)
We study generalized bootstrap confidence regions for the mean of a random vector whose coordinates have an unknown dependency structure. The random vector is supposed to be either Gaussian or to have a symmetric and bounded distribution. The dimensionality
A generalized SidakHolm procedure and control of genralized error rates under independence.
 Statistical Applications in Genetics and Molecular Biology ,
, 2007
"... Abstract For testing multiple null hypotheses, the classical approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate (FWER), the probability of even one false rejection. In many applications, one might be willing to tolerate mo ..."
Abstract

Cited by 13 (4 self)
 Add to MetaCart
(Show Context)
Abstract For testing multiple null hypotheses, the classical approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate (FWER), the probability of even one false rejection. In many applications, one might be willing to tolerate more than one false rejection provided the number of such cases is controlled, thereby increasing the ability of the procedure to detect false null hypotheses. This suggests replacing control of the FWER by controlling the probability of k or more false rejections, which is called the kFWER. In
Asymptotic properties of false discovery rate controlling procedures under independence. ArXiv preprint math.ST/0803.2111v1
, 2008
"... ..."
(Show Context)
Avoiding ‘data snooping’ in multilevel and mixed effects models
 Journal of the Royal Statistical Society Series A (Statistics in Society
, 2007
"... Summary. Multilevel or mixed effects models are commonly applied to hierarchical data. The level 2 residuals, which are otherwise known as random effects, are often of both substantive and diagnostic interest. Substantively, they are frequently used for institutional comparisons or rankings. Diagnos ..."
Abstract

Cited by 7 (0 self)
 Add to MetaCart
Summary. Multilevel or mixed effects models are commonly applied to hierarchical data. The level 2 residuals, which are otherwise known as random effects, are often of both substantive and diagnostic interest. Substantively, they are frequently used for institutional comparisons or rankings. Diagnostically, they are used to assess the model assumptions at the group level. Inference on the level 2 residuals, however, typically does not account for ‘data snooping’, i.e. for the harmful effects of carrying out a multitude of hypothesis tests at the same time. We provide a very general framework that encompasses both of the following inference problems: inference on the ‘absolute ’ level 2 residuals to determine which are significantly different from 0, and inference on any prespecified number of pairwise comparisons. Thus, the user has the choice of testing the comparisons of interest. As our methods are flexible with respect to the estimation method that is invoked, the user may choose the desired estimation method accordingly. We demonstrate the methods with the London education authority data, the wafer data and the National Educational Longitudinal Study data.
Stepdown procedures controlling a generalized false discovery rate. Unpublished manuscript, available at http://astro.temple.edu/ sanat /reports.html
, 2008
"... Abstract: Procedures controlling error rates measuring at least k false rejections, instead of at least one, can potentially increase the ability of a procedure to detect false null hypotheses in situations where one seeks to control k or more false rejections having tolerated a few of them. The kF ..."
Abstract

Cited by 6 (6 self)
 Add to MetaCart
Abstract: Procedures controlling error rates measuring at least k false rejections, instead of at least one, can potentially increase the ability of a procedure to detect false null hypotheses in situations where one seeks to control k or more false rejections having tolerated a few of them. The kFWER, the probability of at least k false rejections, is such an error rate that is recently introduced in the literature and procedures controlling it have been proposed. Recently, Sarkar (2007) introduced an alternative, less conservative notion of error rate, the kFDR, generalizing the usual notion of false discovery rate (FDR), and proposed a procedure controlling it based on kdimensional joint distributions of the null pvalues and assuming the MTP2 (multivariate totally positive of order two) positive dependence among all the pvalues. In this article, we assume a less restrictive form of positive dependence than the MTP2 and develop procedure based only on the bivariate, rather than the kdimensional, distributions of the null pvalues. Key words and phrases: Arbitrary dependence, average power, clumpy dependence, gene expression, generalized FDR, multiple hypothesis testing, positive regression dependence on subset, stepwise procedure. 1
SAMPLE SIZE AND POSITIVE FALSE DISCOVERY RATE CONTROL FOR MULTIPLE TESTING
, 2007
"... Positive false discovery rate (pFDR) is a useful overall measure of errors for multiple hypothesis testing, especially when the underlying goal is to attain one or more discoveries. Control of pFDR critically depends on how much evidence is available from data to distinguish between false and true n ..."
Abstract

Cited by 6 (3 self)
 Add to MetaCart
(Show Context)
Positive false discovery rate (pFDR) is a useful overall measure of errors for multiple hypothesis testing, especially when the underlying goal is to attain one or more discoveries. Control of pFDR critically depends on how much evidence is available from data to distinguish between false and true nulls. Oftentimes, as many aspects of the data distributions are unknown, one may not be able to obtain strong enough evidence from the data for pFDR control. This raises the question as to how much data is needed in order to attain a target pFDR level. We study the asymptotics of the minimum number of observations per null for the pFDR control associated with multiple Studentized tests and F tests, especially when the differences between false nulls and true nulls are small. For Studentized tests, we consider tests on shifts or other parameters associated with normal and general distributions. For F tests, we also take into account the effect of the number of covariates in linear regression. The results show that in determining the minimum sample size per null for pFDR control, higher order statistical properties of data are important, and the number of covariates is important in tests to detect regression effects. 1. Introduction. A
Resamplingbased confidence regions and multiple tests for a correlated random vector
 In Learning Theory: 20th Annual Conference on Learning Theory, COLT 2007
, 2007
"... Abstract. We study generalized bootstrapped confidence regions for the mean of a random vector whose coordinates have an unknown dependence structure, with a nonasymptotic control of the confidence level. The random vector is supposed to be either Gaussian or to have a symmetric bounded distributio ..."
Abstract

Cited by 5 (0 self)
 Add to MetaCart
(Show Context)
Abstract. We study generalized bootstrapped confidence regions for the mean of a random vector whose coordinates have an unknown dependence structure, with a nonasymptotic control of the confidence level. The random vector is supposed to be either Gaussian or to have a symmetric bounded distribution. We consider two approaches, the first based on a concentration principle and the second on a direct boostrapped quantile. The first one allows us to deal with a very large class of resampling weights while our results for the second are restricted to Rademacher weights. However, the second method seems more accurate in practice. Our results are motivated by multiple testing problems, and we show on simulations that our procedures are better than the Bonferroni procedure (union bound) as soon as the observed vector has sufficiently correlated coordinates. 1