Results 1  10
of
34
Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi
"... Does psi exist? In a recent article, Dr. Bem conducted nine studies with over a thousand participants in an attempt to demonstrate that future events retroactively affect people’s responses. Here we discuss several limitations of Bem’s experiments on psi; in particular, we show that the data analysi ..."
Abstract

Cited by 52 (9 self)
 Add to MetaCart
Does psi exist? In a recent article, Dr. Bem conducted nine studies with over a thousand participants in an attempt to demonstrate that future events retroactively affect people’s responses. Here we discuss several limitations of Bem’s experiments on psi; in particular, we show that the data analysis was partly exploratory, and that onesided pvalues may overstate the statistical evidence against the null hypothesis. We reanalyze Bem’s data using a default Bayesian ttest and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem’s pvalues do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
Statistical evidence in experimental psychology: An empirical comparison using 855 t tests
 Perspectives on Psychological Science
, 2011
"... Statistical inference in psychology has traditionally relied heavily on pvalue significance testing. This approach to drawing conclusions from data, however, has been widely criticized, and two types of remedies have been advocated. The first proposal is to supplement p values with complementary me ..."
Abstract

Cited by 27 (4 self)
 Add to MetaCart
Statistical inference in psychology has traditionally relied heavily on pvalue significance testing. This approach to drawing conclusions from data, however, has been widely criticized, and two types of remedies have been advocated. The first proposal is to supplement p values with complementary measures of evidence, such as effect sizes. The second is to replace inference with Bayesian measures of evidence, such as the Bayes factor. The authors provide a practical comparison of p values, effect sizes, and default Bayes factors as measures of statistical evidence, using 855 recently published t tests in psychology. The comparison yields two main results. First, although p values and default Bayes factors almost always agree about what hypothesis is better supported by the data, the measures often disagree about the strength of this support; for 70 % of the data sets for which the p value falls between.01 and.05, the default Bayes factor indicates that the evidence is only anecdotal. Second, effect sizes can provide additional evidence to p values and default Bayes factors. The authors conclude that the Bayesian approach is comparatively prudent, preventing researchers from overestimating the evidence in favor of an effect. Keywords
Default Bayes factors for ANOVA designs.
 Journal of Mathematical Psychology,
, 2012
"... Abstract Bayes factors have been advocated as superior to pvalues for assessing statistical evidence in data. Despite the advantages of Bayes factors and the drawbacks of pvalues, inference by pvalues is still nearly ubiquitous. One impediment to adoption of Bayes factors is a lack of practical ..."
Abstract

Cited by 25 (4 self)
 Add to MetaCart
(Show Context)
Abstract Bayes factors have been advocated as superior to pvalues for assessing statistical evidence in data. Despite the advantages of Bayes factors and the drawbacks of pvalues, inference by pvalues is still nearly ubiquitous. One impediment to adoption of Bayes factors is a lack of practical development, particularly a lack of readytouse formulas and algorithms. In this paper, we discuss and expand a set of default Bayes factor tests for ANOVA designs. These tests are based on multivariate generalizations of Cauchy priors on standardized effects, and have the desirable properties of being invariant with respect to linear transformations of measurement units. Moreover, these Bayes factors are computationally convenient, and straightforward sampling algorithms are provided. We cover models with fixed, random, and mixed effects, including random interactions, and do so for withinsubject, betweensubject, and mixed designs. We extend the discussion to regression models with continuous covariates. We also discuss how these Bayes factors may be applied in nonlinear settings, and show how they are useful in differentiating between the power law and the exponential law of skill acquisition. In sum, the current development makes the computation of Bayes factors straightforward for the vast majority of designs in experimental psychology. * Correspondence: 210 McAlester Hall, Columbia, MO 65203, rouderj@missouri.edu. We thank Brandon Turner and EricJan Wagenmakers for detailed and constructive comments. This research is supported by NSF SES 1024080.
Hierarchical Bayesian parameter estimation for cumulative prospect theory
, 2011
"... a b s t r a c t Cumulative prospect theory (CPT Tversky & Kahneman, 1992) has provided one of the most influential accounts of how people make decisions under risk. CPT is a formal model with parameters that quantify psychological processes such as loss aversion, subjective values of gains and ..."
Abstract

Cited by 14 (2 self)
 Add to MetaCart
(Show Context)
a b s t r a c t Cumulative prospect theory (CPT Tversky & Kahneman, 1992) has provided one of the most influential accounts of how people make decisions under risk. CPT is a formal model with parameters that quantify psychological processes such as loss aversion, subjective values of gains and losses, and subjective probabilities. In practical applications of CPT, the model's parameters are usually estimated using a singleparticipant maximum likelihood approach. The present study shows the advantages of an alternative, hierarchical Bayesian parameter estimation procedure. Performance of the procedure is illustrated with a parameter recovery study and application to a real data set. The work reveals that without particular constraints on the parameter space, CPT can produce loss aversion without the parameter that has traditionally been associated with loss aversion. In general, the results illustrate that inferences about people's decision processes can crucially depend on the method used to estimate model parameters.
Bayesian parametric estimation of stopsignal reaction time distributions
 Journal of Experimental Psychology: General
, 2013
"... The cognitive concept of response inhibition can be measured with the stopsignal paradigm. In this paradigm, participants perform a 2choice response time (RT) task where, on some of the trials, the primary task is interrupted by a stop signal that prompts participants to withhold their response. T ..."
Abstract

Cited by 7 (4 self)
 Add to MetaCart
The cognitive concept of response inhibition can be measured with the stopsignal paradigm. In this paradigm, participants perform a 2choice response time (RT) task where, on some of the trials, the primary task is interrupted by a stop signal that prompts participants to withhold their response. The dependent variable of interest is the latency of the unobservable stop response (stopsignal reaction time, or SSRT). Based on the horse race model (Logan & Cowan, 1984), several methods have been developed to estimate SSRTs. None of these approaches allow for the accurate estimation of the entire distribution of SSRTs. Here we introduce a Bayesian parametric approach that addresses this limitation. Our method is based on the assumptions of the horse race model and rests on the concept of censored distributions. We treat response inhibition as a censoring mechanism, where the distribution of RTs on the primary task (go RTs) is censored by the distribution of SSRTs. The method assumes that go RTs and SSRTs are exGaussian distributed and uses Markov chain Monte Carlo sampling to obtain posterior distributions for the model parameters. The method can be applied to individual as well as hierarchical data structures. We present the results of a number of parameter recovery and robustness studies and apply our approach to published data from a stopsignal experiment.
Why psychologists must change the way they analyze their data: The case of psi
 Journal of Personality and Social Psychology
, 2011
"... Does psi exist? D. J. Bem (2011) conducted 9 studies with over 1,000 participants in an attempt to demonstrate that future events retroactively affect people’s responses. Here we discuss several limitations of Bem’s experiments on psi; in particular, we show that the data analysis was partly explora ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
(Show Context)
Does psi exist? D. J. Bem (2011) conducted 9 studies with over 1,000 participants in an attempt to demonstrate that future events retroactively affect people’s responses. Here we discuss several limitations of Bem’s experiments on psi; in particular, we show that the data analysis was partly exploratory and that onesided p values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem’s data with a default Bayesian t test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem’s p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
The effect of horizontal eye movements on free recall: A preregistered adversarial collaboration
 Journal of Experimental Psychology: General
, 2015
"... A growing body of research has suggested that horizontal saccadic eye movements facilitate the retrieval of episodic memories in free recall and recognition memory tasks. Nevertheless, a minority of studies have failed to replicate this effect. This article attempts to resolve the inconsistent resul ..."
Abstract

Cited by 2 (0 self)
 Add to MetaCart
(Show Context)
A growing body of research has suggested that horizontal saccadic eye movements facilitate the retrieval of episodic memories in free recall and recognition memory tasks. Nevertheless, a minority of studies have failed to replicate this effect. This article attempts to resolve the inconsistent results by introducing a novel variant of proponentskeptic collaboration. The proposed approach combines the features of adversarial collaboration and purely confirmatory preregistered research. Prior to data collection, the adversaries reached consensus on an optimal research design, formulated their expectations, and agreed to submit the findings to an academic journal regardless of the outcome. To increase transparency and secure the purely confirmatory nature of the investigation, the 2 parties set up a publicly available adversarial collaboration agreement that detailed the proposed design and all foreseeable aspects of the data analysis. As anticipated by the skeptics, a series of Bayesian hypothesis tests indicated that horizontal eye movements did not improve free recall performance. The skeptics suggested that the nonreplication may partly reflect the use of suboptimal and questionable research practices in earlier eye movement studies. The proponents countered this suggestion and used a p curve analysis to argue that the effect of horizontal eye movements on explicit memory did not merely reflect selective reporting.
Model comparison and the principle of parsimony
 In
, 2015
"... At its core, the study of psychology is concerned with the discovery of plausible explanations for human behavior. For instance, one may observe that “practice makes perfect”: as people become more familiar with a task, they tend to execute it more quickly and with fewer errors. More interesting is ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
(Show Context)
At its core, the study of psychology is concerned with the discovery of plausible explanations for human behavior. For instance, one may observe that “practice makes perfect”: as people become more familiar with a task, they tend to execute it more quickly and with fewer errors. More interesting is the observation that practice tends to improve
Moving beyond qualitative evaluations of Bayesian models of cognition. Psychonomic bulletin & review
, 2014
"... Abstract Bayesian models of cognition provide a powerful way to understand the behavior and goals of individuals from a computational point of view. Much of the focus in the Bayesian cognitive modeling approach has been on qualitative model evaluations, where predictions from the models are compare ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
(Show Context)
Abstract Bayesian models of cognition provide a powerful way to understand the behavior and goals of individuals from a computational point of view. Much of the focus in the Bayesian cognitive modeling approach has been on qualitative model evaluations, where predictions from the models are compared to data that is often averaged over individuals. In many cognitive tasks, however, there are pervasive individual differences. We introduce an approach to directly infer individual differences related to subjective mental representations within the framework of Bayesian models of cognition. In this approach, Bayesian data analysis methods are used to estimate cognitive parameters and motivate the inference process within a Bayesian cognitive model. We illustrate this integrative Bayesian approach on a model of memory. We apply the model to behavioral data from a memory experiment involving the recall of heights of people. A crossvalidation analysis shows that the Bayesian memory model with inferred subjective priors predicts withheld data better than a Bayesian model where the priors are based on environmental statistics. In addition, the model with inferred priors at the individual subject level led to the best overall generalization performance, suggesting that individual differences are important to consider in Bayesian models of cognition.