Citation Context

...ed on demand. A novice is, by definition, unable to invoke the implicit criteria for making refined judgments about quality. Knowledge of the criteria is "caught" through experience, not defined. It is developed through an inductive process which involves prolonged engagement in evaluative activity shared with and under the tutelage of a person who is already something of a connoisseur. By so doing "the apprentice unconsciously picks up the rules of the art, including those which are not explicitly known to the master... Connoisseurship... can be communicated only by example, not by precept" (Polanyi, 1962, p. 53-54). In other words, providing guided but direct and authentic evaluative experience for students enables them to develop their evaluative knowledge, thereby bringing them within the guild of people who are able to determine quality using multiple criteria. It also enables transfer of some of the responsibility for making evaluative decisions from teacher to learner. In this way, students are gradually exposed to the full set of criteria and the rules for using them, and so build up a body of evaluative knowledge. It also makes them aware of the difficulties which even teachers face of...

Citation Context

...ssessing several classes, but the criteria may require different interpretations, or differ in relative significance, from class to class. The ability to make global appraisals is therefore fundamental to understanding the nature of different classes, and hence to producing something within a particular class. Secondly, something may apparently meet requirements on all appropriate criteria taken individually yet be unsatisfactory overall. It may be difficult to explain this anomaly to students, unless students themselves are confronted with the same evaluative problem. In a different context, Tversky (1969) suggested a line of argument which is perhaps helpful here. Suppose there exists some maximum deficit that could be tolerated on a single criterion before it would be noticed that the expectation had not been met. If on each one of a set of criteria the deficit is less than the tolerable limit, and if there are a number such criteria, the global assessment actually fails the minimum-quality test by an amount equal to the sum of the individual deficits. The global shortfall may be noticeable but not the individual 137 shortfalls. The disqualification is then due less to a single identifiable c...

Citation Context

...d satisfactorily simply through being told about them. Most require practice in a supportive environment which incorporates feedback loops. This usually includes a teacher who knows which skills are to be learned, and who can recognize and describe a fine performance, demonstrate a fine performance, and indicate how a poor performance can be improved. Feedback can also be defined in terms of its effect rather than its informational content: "Feedback is information about the gap between the actual level and the reference level of a system parameter which is used to alter the gap in some way" (Ramaprasad, 1983, p. 4). This alternative definition emphasizes the system-control function. Broadly speaking, feedback provides for two main audiences, the teacher and the student. Teachers use feedback to make programmatic decisions with respect to readiness, diagnosis and remediation. Students use 121 it to monitor the strengths and weaknesses of their performances, so that aspects associated with success or high quality can be recognized and reinforced, and unsatisfactory aspects modified or improved. An important feature of Ramaprasad's definition is that information about the gap between actual and refe...

Citation Context

...iteria should be used (and how), which of the techniques has the soundest theoretical foundation (such as a theory of composition), or which produces the best agreement among competent judges (reliability considerations). An alternative criterion for adjudicating among assessment approaches is the extent to which students improve either as consumers of assessments arrived at by different methods, or through being trained to use a particular assessment approach themselves. With respect to the teaching of writing, these issues have not been thoroughly explored, although they are touched upon by Cooper (1977), Odell and Cooper (1980) and several others. While the line of development in this article is different from that in the literature on writing assessment, it shares an interest in learning outcomes which are complex in the sense that qualitative judgments (defined below) are invariably involved in appraising a student's performance. In such learnings, student development is multidimensional rather than sequential, and prerequisite learnings cannot be conceptualized as neatly packaged units of skills or knowledge. Growth takes place on many interrelated fronts at once and is continuous rather ...

Citation Context

..., student development is multidimensional rather than sequential, and prerequisite learnings cannot be conceptualized as neatly packaged units of skills or knowledge. Growth takes place on many interrelated fronts at once and is continuous rather than lockstep. The outcomes are not easily characterized as correct or incorrect, and it is more appropriate to think in terms of the quality of a student's response or the degree of expertise than in terms of facts memorized, concepts acquired or content mastered. 124 Qualitative judgments defined and characterized A qualitative judgment is defined (Sadler, 1987) as one made directly by a person, the person's brain being both the source and the instrument for the appraisal. Such a judgment is not reducible to a formula which can be applied by a non-expert. In general, qualitative judgments have some or all of the following five characteristics: 1. Multiple criteria are used in appraising the quality of performances. As well as the individual dimensions represented by the criteria, the total pattern of relationships among those dimensions is important. In this sense the criteria interlock, so that the overall configuration amounts to more than the sum ...

Citation Context

...interpretation of scores. Only cursory attention has usually been given to feedback and formative assessment, and then it is mostly hortatory, recipe-like and atheoretic. In many cases feedback and formative assessment (or their equivalents) are not mentioned at all in either the body of the text or the index, although the books by Rowntree (1977), Bloom, Madaus and Hastings (1981), Black and Dockrell (1984) and Chater (1984) are notable exceptions. In general, a concern with the aims of summative assessment has dominated the field in terms of both research and the guidance given to teachers (Black, 1986). This dominance is implicit in the treatment given, for instance, to reliability and validity. Textbooks almost invariably describe how the validity (of assessments) is to be distinguished from the reliability (of grades or classifications). Reliability is usually (and correctly) said to be a necessary but not sufficient condition for validity, because measurements or judgments may be reliable in the sense of being consistent over time or over judges and still be off-target (or invalid). Reliability is therefore presented as a precondition for a determination of validity. In discussing format...

Citation Context

...to revise and refine their own work in cooperation with the teacher, and by editing and helping other students to improve theirs (Beaven, 1977; Pianko and Radzik, 1980; Thompson, 1981; Chater, 1984). "Students who become conscious of what they're doing by explaining their decisions to other students also learn new strategies for solving writing problems. And because students should become progressively more independent and self-confident as writers, they need to evaluate each other's work and their own frequently, a practice which teaches constructive criticism, close reading, and rewriting" (Lindemann, 1982, p. 234). Boud (1986) reported similar findings in higher education when self-assessment and peer-assessment were built into instructional procedures for law, engineering and architecture students. It is clear that to build explicit provision for evaluative experience into an instructional system enables learners to develop self-assessment skills and gap-closing strategies simultaneously, and therefore to move towards self-monitoring. Some resistance to this proposition can, however, be expected. Factors militating against self-monitoring The lack of opportunities typically given to students ...

Citation Context

... is therefore presented as a precondition for a determination of validity. In discussing formative assessment, however, the relation between reliability and validity is more appropriately stated as follows: validity is a sufficient but not necessary condition for reliability. Attention to the validity of judgments about individual pieces of work should take precedence over attention to reliability of grading in any context where the emphasis is on diagnosis and improvement. Reliability will follow as a corollary. Acceptance of this principle, which is emphasized by only a few writers (such as Nitko, 1983), has implications for how the process of appraisal is conceptualized, and the mechanisms of improvement understood. In the literature on learning research, feedback is usually identified with knowledge of results (often abbreviated to KR), a concept which gained considerable currency through Thorndike's (1913) so-called Law of Effect. Reviewing a series 123 of experimental studies on learning from written materials (texts and programmed instruction), Kulhavy (1977, p. 211) defined feedback as "any of the numerous procedures that are used to tell a learner if an instructional response is right...

Citation Context

...efforts together with the teachers' appraisals of those efforts, simply because the work is produced in workshops, studios, theatres and other open environments. The best examples, or perhaps exemplary material developed outside the classroom, serve naturally and unobtrusively as reference points. In the liberal arts and humanities, however, students often work privately, and do not get to see or read what other students have produced. What constitutes work of high quality then remains to some extent unknown. Exceptional cases aside, it is ironic that the prototypes of competency levels which Myers (1980) recommended as necessary for assessors using holistic methods for the evaluation of writing are not similarly considered a general requirement for students learning to write or to master other complex skills. Standards as goals or aspirations In its simplest form, a standard or reference level is a designated degree of performance or excellence. It becomes a goal when it is desired, aimed for, or aspired to. Some goals are external (assigned by a teacher) while others are developed or adapted by the learners themselves. A learner may decide to ignore or reject an external goal, in which case ...

Citation Context

...eria change from task to task) when returning assessed papers. Teachers who use criterion sheets regularly, however, 134 find that while such sheets are helpful, they may lead to frustration because of their inflexibility. The qualities of a piece of work cannot necessarily be dealt with adequately using a fixed criterion set, and teachers often feel the need to call upon nonstandard criteria. A more satisfactory (and less mechanistic) solution to the problem is to consider the universe of criteria as notionally partitioned into two subsets called for convenience manifest and latent criteria (Sadler, 1983). Manifest criteria are those which are consciously attended to either while a work is being produced or while it is being assessed. Latent criteria are those in the background, triggered or activated as occasion demands by some (existential) property of the work that deviates from expectation. Whenever there is a serious violation of a latent criterion, the teacher invokes it, and it is added (at least temporarily) to the working set of manifest criteria. This is possible because competent teachers have a thorough knowledge of the full set of criteria, and the (unwritten) rules for using them...

Citation Context

... own work in cooperation with the teacher, and by editing and helping other students to improve theirs (Beaven, 1977; Pianko and Radzik, 1980; Thompson, 1981; Chater, 1984). "Students who become conscious of what they're doing by explaining their decisions to other students also learn new strategies for solving writing problems. And because students should become progressively more independent and self-confident as writers, they need to evaluate each other's work and their own frequently, a practice which teaches constructive criticism, close reading, and rewriting" (Lindemann, 1982, p. 234). Boud (1986) reported similar findings in higher education when self-assessment and peer-assessment were built into instructional procedures for law, engineering and architecture students. It is clear that to build explicit provision for evaluative experience into an instructional system enables learners to develop self-assessment skills and gap-closing strategies simultaneously, and therefore to move towards self-monitoring. Some resistance to this proposition can, however, be expected. Factors militating against self-monitoring The lack of opportunities typically given to students to make appropriate qu...

Citation Context

...ing programs are conceived of as divisible into logically dependent units which can be mastered more or less sequentially, one by one. The resulting technology is associated with test scores, diagnostic items, criterionreferencing and mastery learning. Other lines of research occur in specific subject areas. Of particular interest is the literature on the assessment of writing, which contains descriptions of a number of different approaches, including assessment by means of general impression, analytic scales, primary traits, syntactic features, relative readability and intellectual strategy (Gere, 1980). These differ not only in procedural detail, but also in their theoretical bases. Much of the discussion about and evaluation of the various possibilities has revolved around which assessment criteria should be used (and how), which of the techniques has the soundest theoretical foundation (such as a theory of composition), or which produces the best agreement among competent judges (reliability considerations). An alternative criterion for adjudicating among assessment approaches is the extent to which students improve either as consumers of assessments arrived at by different methods, or th...

Citation Context

...and the complex interrelations which exist among the criteria, it is clear that to use the whole set for a particular assessment would be unmanageable. How judges cope with the situation therefore requires some investigation. The literature on research into human judgmental processes in a variety of settings is both instructive and extensive, and cannot be adequately summarized here. But of particular concern to researchers have been the inefficiency of intuitive judgmental processes, and the limitations in human information processing capacities which result in biased or defective decisions (Sadler, 1981). In broad terms, the many techniques proposed for making complex judgments fall more or less into two camps, each of which has its research tradition, its advocates and its detractors. Fortunately, it not necessary to make a firm decision on one or 132 the other for purposes of formative assessment. Both can be drawn upon because evaluative input can take any appropriate form, and in any case is always open to discussion, clarification, and revision if necessary. The first general line of attack is to devise and implement a procedure which begins with identifying a number of relevant criteria...

Citation Context

...of the judgmental behaviours of competent assessors. Diederich (1974) followed the latter approach. This component-wise attack on the problem of making multicriterion judgments is often advocated as the ideal towards which impressionistic, holistic or informal systems should be made to move. It assumes, however, that the set of criteria nominated is sufficient for all cases, that the criteria do not overlap, and that use of the combining formula leads to judgments which would not conflict (except perhaps rarely) with more holistic approaches. A substantial argument has been mounted elsewhere (Sadler, 1985) that for complex phenomena, use of a fixed set of criteria (and therefore the analytic approach) is potentially limiting. The second approach to making complex judgments is for the evaluator to react to the work as a whole, making an entire, or what Kaplan called a configurational (1964, p. 211), assessment first and then to substantiate it (to whatever extent is necessary) by referring to separate criteria, which may or may not be drawn from a prespecified set. In this approach, imperfectly differentiated criteria are compounded as a kind of gestalt and projected onto a single scale of quali...

Citation Context

...t is, without reference to other students' work). Teachers tacitly 127 acknowledge the difficulty of relying on memory alone when they make a survey of pieces of student work before assigning grades to them. This survey generates a loosely quantitative baseline or frame of reference for what is to be regarded as barely satisfactory and what is to count as excellent in the context. Even after a survey has been made, however, smaller scale order effects (especially severity, leniency, and carryover) almost invariably occur. This is a subject of continuing research (see, for example, the work of Hales and Tokar, 1975, and Daly and Dickson-Markman, 1982) and can be interpreted in terms of Helson's (1959) adaptation level theory. It therefore appears that teachers' conceptions of quality and standards exist in some quiescent and pliable form until they are reconstituted by fresh evaluative activity. In an instructional system, an exclusive reliance on teachers' guild knowledge works against the interests of the learner in two important ways. In the first place, although the practice of surveying a sample of performances is common (and advisable where the aim is fair ranking of one student's work against tha...

Citation Context

...rement and assessment published during the past 25 years have placed great emphasis on achieving high content validity in teachermade tests, producing reliable scores or grades, and the statistical manipulation or interpretation of scores. Only cursory attention has usually been given to feedback and formative assessment, and then it is mostly hortatory, recipe-like and atheoretic. In many cases feedback and formative assessment (or their equivalents) are not mentioned at all in either the body of the text or the index, although the books by Rowntree (1977), Bloom, Madaus and Hastings (1981), Black and Dockrell (1984) and Chater (1984) are notable exceptions. In general, a concern with the aims of summative assessment has dominated the field in terms of both research and the guidance given to teachers (Black, 1986). This dominance is implicit in the treatment given, for instance, to reliability and validity. Textbooks almost invariably describe how the validity (of assessments) is to be distinguished from the reliability (of grades or classifications). Reliability is usually (and correctly) said to be a necessary but not sufficient condition for validity, because measurements or judgments may be reliable i...

Citation Context

...other students' work). Teachers tacitly 127 acknowledge the difficulty of relying on memory alone when they make a survey of pieces of student work before assigning grades to them. This survey generates a loosely quantitative baseline or frame of reference for what is to be regarded as barely satisfactory and what is to count as excellent in the context. Even after a survey has been made, however, smaller scale order effects (especially severity, leniency, and carryover) almost invariably occur. This is a subject of continuing research (see, for example, the work of Hales and Tokar, 1975, and Daly and Dickson-Markman, 1982) and can be interpreted in terms of Helson's (1959) adaptation level theory. It therefore appears that teachers' conceptions of quality and standards exist in some quiescent and pliable form until they are reconstituted by fresh evaluative activity. In an instructional system, an exclusive reliance on teachers' guild knowledge works against the interests of the learner in two important ways. In the first place, although the practice of surveying a sample of performances is common (and advisable where the aim is fair ranking of one student's work against that of other students), it is inappropr...

Citation Context

...e used (and how), which of the techniques has the soundest theoretical foundation (such as a theory of composition), or which produces the best agreement among competent judges (reliability considerations). An alternative criterion for adjudicating among assessment approaches is the extent to which students improve either as consumers of assessments arrived at by different methods, or through being trained to use a particular assessment approach themselves. With respect to the teaching of writing, these issues have not been thoroughly explored, although they are touched upon by Cooper (1977), Odell and Cooper (1980) and several others. While the line of development in this article is different from that in the literature on writing assessment, it shares an interest in learning outcomes which are complex in the sense that qualitative judgments (defined below) are invariably involved in appraising a student's performance. In such learnings, student development is multidimensional rather than sequential, and prerequisite learnings cannot be conceptualized as neatly packaged units of skills or knowledge. Growth takes place on many interrelated fronts at once and is continuous rather than lockstep. The outcom...

Citation Context

...d during the past 25 years have placed great emphasis on achieving high content validity in teachermade tests, producing reliable scores or grades, and the statistical manipulation or interpretation of scores. Only cursory attention has usually been given to feedback and formative assessment, and then it is mostly hortatory, recipe-like and atheoretic. In many cases feedback and formative assessment (or their equivalents) are not mentioned at all in either the body of the text or the index, although the books by Rowntree (1977), Bloom, Madaus and Hastings (1981), Black and Dockrell (1984) and Chater (1984) are notable exceptions. In general, a concern with the aims of summative assessment has dominated the field in terms of both research and the guidance given to teachers (Black, 1986). This dominance is implicit in the treatment given, for instance, to reliability and validity. Textbooks almost invariably describe how the validity (of assessments) is to be distinguished from the reliability (of grades or classifications). Reliability is usually (and correctly) said to be a necessary but not sufficient condition for validity, because measurements or judgments may be reliable in the sense of bei...

Citation Context

...ment in the literature Authors of textbooks on measurement and assessment published during the past 25 years have placed great emphasis on achieving high content validity in teachermade tests, producing reliable scores or grades, and the statistical manipulation or interpretation of scores. Only cursory attention has usually been given to feedback and formative assessment, and then it is mostly hortatory, recipe-like and atheoretic. In many cases feedback and formative assessment (or their equivalents) are not mentioned at all in either the body of the text or the index, although the books by Rowntree (1977), Bloom, Madaus and Hastings (1981), Black and Dockrell (1984) and Chater (1984) are notable exceptions. In general, a concern with the aims of summative assessment has dominated the field in terms of both research and the guidance given to teachers (Black, 1986). This dominance is implicit in the treatment given, for instance, to reliability and validity. Textbooks almost invariably describe how the validity (of assessments) is to be distinguished from the reliability (of grades or classifications). Reliability is usually (and correctly) said to be a necessary but not sufficient condition for...

Citation Context

...nstructively appraising the work of fellow learners is already established as part of normal teaching in some subjects and fields. Many teachers, for example, encourage their students to exchange work with one another in class. In particular, these principles are foundational to certain approaches to the teaching of writing, specifically reader-writer conferencing, peer review, and process writing. Students develop their pool of strategies by learning to revise and refine their own work in cooperation with the teacher, and by editing and helping other students to improve theirs (Beaven, 1977; Pianko and Radzik, 1980; Thompson, 1981; Chater, 1984). "Students who become conscious of what they're doing by explaining their decisions to other students also learn new strategies for solving writing problems. And because students should become progressively more independent and self-confident as writers, they need to evaluate each other's work and their own frequently, a practice which teaches constructive criticism, close reading, and rewriting" (Lindemann, 1982, p. 234). Boud (1986) reported similar findings in higher education when self-assessment and peer-assessment were built into instructional procedures f...

Citation Context

...he work of fellow learners is already established as part of normal teaching in some subjects and fields. Many teachers, for example, encourage their students to exchange work with one another in class. In particular, these principles are foundational to certain approaches to the teaching of writing, specifically reader-writer conferencing, peer review, and process writing. Students develop their pool of strategies by learning to revise and refine their own work in cooperation with the teacher, and by editing and helping other students to improve theirs (Beaven, 1977; Pianko and Radzik, 1980; Thompson, 1981; Chater, 1984). "Students who become conscious of what they're doing by explaining their decisions to other students also learn new strategies for solving writing problems. And because students should become progressively more independent and self-confident as writers, they need to evaluate each other's work and their own frequently, a practice which teaches constructive criticism, close reading, and rewriting" (Lindemann, 1982, p. 234). Boud (1986) reported similar findings in higher education when self-assessment and peer-assessment were built into instructional procedures for law, engineer...

Citation Context

...sely to learn that there are different ways in which work of a particular quality can find expression. There is often a wide variety of objects within the same genre which are regarded as excellent. Unless students come to this understanding, and learn how to abstract the qualities which run across cases with different surface features but which are judged equivalent, they can hardly be said to appreciate the concept of quality at all. The second consideration is that originality and creativity are not usually, contrary to some opinion, best developed in a completely freewheeling environment. Bailin (1987) pointed out that there is no essential conflict between creative processes and the production of something which is generally accepted as of high quality. Creative productions are mostly highly disciplined, and are almost invariably produced not by accident or through random risk taking but when the producer, by being thoroughly conversant with the characteristics of the discipline or genre, understands when and how to transcend the normal boundaries. Knowing the metacriteria, that is, knowing when the suspension of some criterion, even on occasion a principal one, can be justified in favour ...

Citation Context

...sible because competent teachers have a thorough knowledge of the full set of criteria, and the (unwritten) rules for using them. But it is precisely this type of knowledge which must be developed within the students if they are to be able to monitor their own performances with a reasonable degree of sophistication. The translation of a criterion from latent to manifest should therefore not be interpreted by either the student or the teacher as unfair or as some sort of aberration. Because of the practical impossibility of employing all criteria at once, it is inevitable and perfectly normal. Marshall (1958, 1968) referred to this as the flotation principle, and advocated its use in evaluation. In an interesting shift of metaphor, it also formed the basis for Elbow's (1973) so-called center of gravity approach to appraising student writing for formative purposes. The art of formative assessment is to generate an efficient and partly reversible progression in which criteria are translated for the student's benefit from latent to manifest and back to latent again. The aim is to work towards ultimate submergence of many of the routine criteria once they are so obviously taken for granted that they ...

Citation Context

...improvements made in some directions may expose residual (or even precipitate new) shortcomings in other directions. For these reasons it would be difficult if not impossible in the situations described above to automate or develop a computer-based system for feedback or formative assessment, or for generating remedial moves and appropriate corrective procedures. Any attempt to mechanize such educational activities and creative efforts is unlikely to be successful because of the large number of variables involved, the intense relations often existing among them, and their essential fuzziness (Sadler, 1982). But the inability to mechanize a system that ordinarily depends heavily on qualitative judgments does not, of course, mean that such a system cannot be made to work. People frequently not only make, share, and broadly agree on qualitative judgments, but also use them as the basis for their own improvement. By definition, something which can be shown to occur is more than just a theoretical possibility, and it is common knowledge that a complex activity can be subject to a high degree of control even when the individual processes have not been comprehensively analyzed and are not fully unders...