Test statistics that are suitable for testing composite hypotheses are typically non-pivotal, and conservative bounds are commonly used to test composite hypotheses. In this paper, we propose a testing procedure for composite hypotheses that incorporates additional sample information. This avoids, as n →∞, the use of conservative bounds and leads to tests with better power than standard tests. The testing procedure satisfies a novel similarity condition that is relevant for asymptotic tests of composite hypotheses, and we show that this is a necessary condition for a test to be unbiased. The procedure is particularly useful for simultaneous testing of multiple inequalities, in particular when the number of inequalities is large. This is the situation for the multiple comparisons of forecasting models, and we show that the new testing procedure dominates the ‘reality check ’ of White (2000) and avoids certain pitfalls.

These days, statisticians often deal with complex, high dimensional datasets. Researchers in statistics and machine learning have responded by creating many new methods for analyzing high dimensional data. However, many of these new methods depend on strong assumptions. The challenge of bringing low assumption inference to high dimensional settings requires new ways to think about the foundations of statistics. Traditional foundational concerns, such as the Bayesian versus frequentist debate, have become less important. 1. In the Olden Days There is a joke about media bias from the comedian Al Franken: “To make the argument that the media has a left- or right-wing, or a liberal or a conservative bias, is like asking if the problem with Al-Qaeda is: do they use too much oil in their hummus?” I think a similar comment could be applied to the usual debates in the foundations of statistical inference. The important foundation questions are not ‘Bayes versus Frequentist ’ or ‘Objective Bayesian versus Subjective Bayesian’.