lic Services in Rural.. for Rural DevelOpmentf,---419p.; Monograph composed o papers presented at conference developed by the North Central Regional Center for Rural Development (Lincoln, Nebraska)

VARCOMP. Generalizability theory (Brennan, 1983; Cronbach, Gleser, Nanda, & Raja-ratnam, 1972) is potentially a very powerful methodology for evaluating educational tests. Its use, however, has hitherto been limited because of the lack of suitable software for undertaking such data analyses. Although

Abstract Item response theory is usually applied to items with a selected-response format, such as multiple choice items, whereas generalizability theory is usually applied to constructed-response tasks assessed by raters. However, in many situations, raters may use rating scales consisting

Statistical theories of measurement scores have evolved over the last 100 years. In 1904,

In recent years, measures of effect size have become central to many areas of psychological research. Many argue that reporting effect sizes is essential due to the inherent problems with statistical significance testing (Wilkinson et al., 1999). In addition, effect size estimates serve as the basic data for meta-analysis, and therefore are critical for modern literature reviews. Accurate effect size estimates are also needed for effective power analysis, which plays an important role in the design of new research. Effect size estimates will be useful only to the extent that they can be interpreted beyond the scope of an individual study. Unfortunately, studies that utilize different research designs will often produce effect sizes that estimate different parameters, and therefore are not directly comparable. Several aspects of the study design will impact the comparability of the effect sizes, such as the choice of an outcome measure, the strength of experimental manipulations, and the control for confounding factors. In this paper, we focus on only one of these issues-- the impact of the study design on the metric of the effect size estimate. In order for effect size estimates to be comparable across designs, they must be expressed in a common metric. For example, if different studies use different measures for the dependent variable, the mean difference between groups will not be directly comparable, unless the effect size is defined in a

Abstract This article introduces the application of Generalizability Theory to assessing the reliability of measurements. Generalizability Theory can be used to assess reliability in the presence of multiple sources of error and also can be applied to assess reliability in the presence

Many annotation projects have shown that the quality of manual annotations often is not as good as would be desirable for reliable data analysis. Identifying the main sources responsible for poor annotation quality must thus be a major concern. Generalizability theory is a valuable tool

in the following collection(s): This article, along with others on similar topics, appears e-Letters "Responses " in the online version of this article. "Submit a response " in the right-hand menu under or click onhere To submit an e-Letter on this article, click E-mail alerts to receive free e-mail alerts hereSign up by guest on May 17,

This paper discusses the benefits of using generalizabilty theory in lieu of classical test theory. Generalizability theory subsumes and extends the precepts of classical test theory by estimating the magnitude of multiple sources of measurement error and their interactions simultaneously in a

. In this article, we review some debates about the use of Generalizability Theory in esti-mating reliability of single-subject observational data. We propose that it could be used but under a clearly stated set of conditions. The conceptualization of facets and object of measurement for a common design