This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff’s alpha as well as Scott’s pi and Cohen’s kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in Computational Linguistics, may be more appropriate for many corpus annotation tasks – but that their use makes the interpretation of the value of the coefficient even harder.

Collaborative interactions with many existing digital tabletop systems lack the fluidity of collaborating around a table using traditional media. This paper presents a critical analysis of the current state-of-the-art in digital tabletop systems research, targeted at discovering how user requirements for collaboration are currently being met and uncovering areas requiring further development. By considering research on tabletop displays, collaboration, and communication, several design guidelines for effective colocated collaboration around a tabletop display emerged. These guidelines suggest that technology must support: (1) natural interpersonal interaction, (2) transitions between activities, (3) transitions between personal and group work, (4) transitions between tabletop collaboration and external work, (5) the use of physical objects, (6) accessing shared physical and digital objects, (7) flexible user arrangements, and (8) simultaneous user interactions. The critical analysis also revealed several important directions for future research, including: standardization of methods to evaluate co-located collaboration; comparative studies to determine the impact of existing system configurations on collaboration; and creation of a taxonomy of collaborative tasks to help determine which tasks and activities are suitable for tabletop collaboration.

We describe our experience in developing a discourse-annotated corpus for community-wide use. Working in

This thesis is an inquiry into the nature of the high-level, rhetorical structure of unrestricted natural language texts, computational means to enable its derivation, and two applications (in automatic summarization and natural language generation) that follow from the ability to build such structures automatically. The thesis proposes a first-order formalization of the high-level, rhetorical structure of text. The formalization assumes that text can be sequenced into elementary units; that discourse relations hold between textual units of various sizes; that some textual units are more important to the writer's purpose than others; and that trees are a good approximation of the abstract structure of text. The formalization also introduces a linguistically motivated compositionality criterion, which is shown to hold for the text structures that are valid. The thesis proposes, analyzes theoretically, and compares empirically four algorithms for determining the valid text structures of ...

Reuters, the global information, news and technology group, has for the first time made available free of charge, large quantities of archived Reuters news stories for use by research communities around the world. The Reuters Corpus Volume 1 (RCV1) includes over 800,000 news stories- typical of the annual English language news output of Reuters. This paper describes the origins of RCV1, the motivations behind its creation, and how it differs from previous corpora. In addition we discuss the system of category coding, whereby each story is annotated for topic, region and industry sector. We also discuss the process by which these codes were applied, and examine the issues involved in maintaining quality and consistency of coding in an operational, commercial environment. 1.

The TIPSTER Text Summarization Evaluation (SUMMAC) has established definitively that automatic text summarization is very effective in relevance as- sessment tasks. Summaries as short as 17% of full text length sped up decision- making by almost a factor of 2 with no statistically significant degradation in F- score acuracy. SUMMAC has also in- troduced a new intrinsic method for automated evaluation of informative sum- maries.

In order to build robust automatic ab- stracting systems, there is a need for better training resources than are currently available. In this paper, we introduce an annotation scheme for scientific ar- ticles which can be used to build such a resource in a consistent way. The seven categories of the scheme are based on rhetorical moves of argumentation.

this paper, we present an implemented system for processing definite Universidade do Vale do Rio dos Sinos - UNISINOS, Av. Unisinos 950 - Cx. Postal 275, 93022-000

We discuss a tagging schema and a tagging tool for labeling the rhetorical structure of texts. We also propose a statistical method for measuring agreement of hierarchical structure annotations and we discuss its strengths and weaknesses. The statistical measure we use suggests that annotators can achieve good levels of agreement on the task of determining the high-level, rhetorical structure of texts. Our empirical experiments also suggest that building discourse parsers that incrementally derive correct rhetorical structures of unrestricted texts without applying any form of backtracking is unfeasible. 1

Abstract not found