Rebecca J. Passonneau and Diane J. Litman. Intention-Based Segmentation: Human Reliability and Correlation with Linguistic Cues. In Proceedings of the 31st Annual Meeting of the ACL, pages 148--155, Columbus, Ohio, June 1993. Association for Computational Linguistics.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....reason for writing sentences 1.3 to 1.5, perhaps to clarify for the reader what was written in 1.2. The details given in 1.3 to 1.5 serve this purpose of the author. There is evidence that people perform this kind of discourse segmentation during understanding. For example, Passonneau and Litman [19] and Hirschberg and Grosz [7] found statistically significant agreement among subjects asked to perform a discourse segmentation task. How might a computer perform such segmentation or produce language from which such segments can be recovered In their study, Hirschberg and Grosz also ....

Passonneau, R. and Litman, D. Intention-based segmentation: Human reliability and correlation with linguistic cues. In Proceedings of the 31st Meeting of the Association for Computational Linguistics, (June 22--26, Columbus, Ohio). Association for Computational Linguistics, N.J., 1993, 148--155.

....of previously occurring objects will be treated as known. Related Work There has been recent growth in empirically oriented work in discourse processing. Several researchers have addressed evaluation, investigating the degree to which human subjects agree with one another on discourse tasks (Passonneau Litman 1993, Hirschberg Grosz 1992, Hearst 1993) Others have used frequency information to evaluate algorithms. Passonneau and Litman (1993) tagged a corpus with classes and features and tested algorithms hypothesized from the literature. They consider just one feature at a time, so do not address ....

....work in discourse processing. Several researchers have addressed evaluation, investigating the degree to which human subjects agree with one another on discourse tasks (Passonneau Litman 1993, Hirschberg Grosz 1992, Hearst 1993) Others have used frequency information to evaluate algorithms. Passonneau and Litman (1993) tagged a corpus with classes and features and tested algorithms hypothesized from the literature. They consider just one feature at a time, so do not address interactions among multiple features. Some researchers have derived algorithms and models on the basis of frequency information, but have ....

Passonneau, R. & Litman, D. 1993. Intention-Based Segmentation: Human Reliability and Correlation with Linguistic Cues. In Procs. 31st Annual Meeting of the Assoc. for Computational Linguistics (ACL93) , pp. 148-155.

....to be continuous if its discontinuity is close to 0. Sum. ratio in word number indicates the average length of summaries when it is calculated in the number of words. We tested the statistical significance of the agreement among subjects. Using the same methodology as in (Jing et al. 1998; Passonneau and Litman, 1993), we performed Cochran s Q test(Degroot et al. 1981) on the data from the subjects. For our task, Cochran s Q test evaluates the null hypothesis that the total number of human subjects judging the same document as relevant is randomly distributed. The results show that this hypothesis is false ....

Passonneau, R.J. and D.J. Litman, 1993. Intention-based Segmentation: Human Reliability and Correlation with Linguistic Cues. In 31st Annual Meeting of the Association for Computational Linguistics. pages 148--155.

....Cornell University, and Beer Sheva University in Israel. 4. 2 Agreement Among Human Subjects We measured agreement among human subjects using percent agreement, a metric defined by [30] for the sense disambiguation task, but also used in other applications such as discourse segmentation [12, 7]. Percent agreement is the ratio of observed agreements with the majority opinion to possible agreements with the majority opinion. For our experiments, agreement among 3 or more subjects is a majority opinion. The total possible agreements with the majority opinion is the number of human subjects ....

....we use an evaluation of 10 and 20 summaries in order to decrease the bias of the length factor. Microsoft Lexical Chain Prec Recall Prec Recall 10 33 37 61 67 20 32 39 47 64 Table 2: Evaluation of summarization programs. 4. 3 Statistical Significance Using the same methodology as in [12, 7, 19], we applied Cochran s test to our data. For our application, Cochran s test evaluates the null hypothesis that the total number of human subjects extracting the same sentence is randomly distributed. Our results show that the agreement among subjects is highly significant. That is, the ....

Passonneau Rebecca J. and Diane J.Litman. Intention-based segmentation: human reliability and correlation with linguistic cues. In Proceedings of the 31th Annual Meeting of the Association for Computational Linguistics (ACL-93), pages 148--155, Ohio, 1993.

....If we discard unfilled hesitations, the judgment of whether or not a repair occurs between two particular words or during a particular dialogue move appears to be sufficiently reliable to use. Reliability is frequently cited as some form of percent agreement (e.g. Passonneau and Litman [14], TOBI [17] Kowtko et al. 8] such as the percentage of judgments on which two coders agreed. in this context, this is misleading because repairs are coded during only approximately 5 of moves and after only approximately 3 of words. For instance, the distribution of agreement rates for two ....

R. J. Passonneau and D. J. Litman. Intention-based segmentation: human reliability and correlation with linguistic cues. In Proceedings of the ACL, pages 148--155, 1993.

....and Beer Sheva University in Israel. Agreement Among Human Subjects We measured agreement among human subjects using percent agreement, a metric defined by (Gale, Church, Yarowsky 1992) for the sense disambiguation task, but also used in other applications such as discourse segmentation (Passonneau Litman 1993; Hearst 1994) Percent agreement is the ratio of observed agreements with the majority opinion to possible agreements with the majority opinion. For our experiments, agreement among 3 or more subjects is a majority opinion. The total possible agreements with the majority opinion is the number of ....

....al. 1997) found that the most important 20 paragraphs extracted by 2 subjects have only 46 overlap. The two most probable reasons for this high percent agreement are the style of the TREC articles and our restriction on uniform length. Statistical Significance Using the same methodology as in (Passonneau Litman 1993; Hearst 1994; Marcu 1997) we applied 1 According to (Jing et al. 1998) ideal summary based evaluation is extremely sensitive to the required summary length. Therefore, we use an evaluation of 10 and 20 summaries in order to decrease the bias of the length factor. Microsoft Lexical Chain Prec ....

Passonneau, R., and Litman, D. 1993. Intentionbased segmentation: Human reliability and correlation with linguistic cues. In Proceedings of the 31th Annual Meeting of the Association for Computational Linguistics (ACL-93), 148--155. Ohio: Association for Computational Linguistics.

....such as the how the current utterance relates to prior discourse. We have identified four types of discourse cues. The first type is perceptible silence, or pauses, observed at the end of an utterance, which has been found to correlate with discourse boundaries (Grosz and Hirschberg, 1992;Passonneau and Litman, 1993; Swerts, 1997) We believe that in the context of initiative modeling, silence at the end of an utterance may suggest that the speaker has nothing more to say in the current turn and intends to give up his task dialogue initiative. For instance, in the following dialogue segment, the silence at ....

Passonneau, R. J. and D. J. Litman: 1993, `Intention-Based Segmentation: Human Reliability and Correlation with Linguistic Cues'. In: Proceedings of the 31st Annual Meeting of the Association for Computational Linguistics. pp. 148--155.

....units 1 and 3 in text (4) are assigned the score 3. Agreement among judges Overall agreement among judges. I measured the agreement of the judges with one another, by means of the notion of percent agreement that was defined by Gale (1992) and used extensively in discourse segmentation studies (Passonneau Litman 1993; Hearst 1997) Percent agreement reflects the ratio of observed to possible agreements with the majority opinion. The percent agreements computed for each of the five texts and each level of importance are given in table 3. The agreements among judges for my experiment seem to follow the same ....

....very important, 63 for those considered less important, and 77 for those considered unimportant. The overall percent agreement in this case is 75 . Statistical significance. It has often been emphasized that agreement figures of the kinds computed above could be misleading (Krippendorff 1980; Passonneau Litman 1993). Since the true set of important textual units cannot be independently known, we cannot compute how valid the importance assignments of the judges were. Moreover, although the agreement figures that would occur by chance offer a strong indication that our data are reliable, they do not provide ....

[Article contains additional citation context not shown here]

Passonneau, R., and Litman, D. 1993. Intention-based segmentation: human reliability and correlation with linguistic cues. In Proceedings of the 31st Annual Meeting of the Association for Computational Linguistics (ACL--93), 148--155.

....University. Results and Analysis Agreement Among Human Subjects We measured agreement among human subjects using percent agreement, a metric defined by (William, Church, Yarowsky 1992) for the word sense disambiguation task, but also used in other applications such as discourse segmentation (Passonneau Litman 1993; Hearst 1994) Percent agreement is the ratio of observed agreements with the majority opinion to possible agreements with the majority opinion. For our Doc. Num 536075 Doc. Title State Sponsored Death Squads Blocking Third World Development sent Sub1 Sub2 Sub3 Sub4 Sub5 SysA SysB SysC num ....

....did not require human judges to create a summary of a given length. Only (Salton et al. 1997) has a strict restriction in terms of length, but the extracted unit is a paragraph instead of a sentence, so the result is not comparable. Statistical Significance Using the same methodology in (Passonneau Litman 1993; Hearst 1994; Marcu 1997) we applied Cochran s test to our data. For our application, Cochran s test evaluates the null hypothesis that the total number of human subjects extracting the same sentence is randomly distributed. Cochran s statistic Q approximates the 2 distribution with j Gamma ....

Passonneau, R. J., and Litman, D. J. 1993. Intentionbased segmentation: human reliability and correlation with linguistic cues. In Proceedings of the 31th Annual Meeting of the Association for Computational Linguistics (ACL-93), 148--155.

....space of anaphoric antecedents to those discourse entities actually referred to in the discourse, while the cache model allows unrestricted retrieval in the main or long term memory. Many studies on discourse segmentation highlight the role of cue words for signaling segment boundaries (cf. e.g. Passonneau Litman (1993)) or the use of overspecified referential expressions to indicate a thematic shift (Vonk et al. 1992; Walker, 1996b) However useful these strategies might be, we see the danger that such a surface level description may actually hide structural regularities at deeper levels of investigation ....

....still has several restrictions. First, it has been developed on the basis of a small corpus of written texts. Though these cover diverse text sorts (viz. technical product reviews, newspaper articles and literary narratives) we currently do not account for spoken monologues as modelled, e.g. by Passonneau Litman (1993) or even the intricacies of dyadic conversations Rose et al. 1995) deal with. Second, a thorough integration of the referential and intentional description fo discourse segments still has to be worked out. Acknowledgments. We like to thank our colleagues in the CLIF group for fruitful ....

Passonneau, R. J. & D. J. Litman (1993). Intention based segmentation: Human reliability and correlation with linguistic cues. In Proc. of the 31 st Annual Meeting of the Association for Computational Linguistics; Columbus, Ohio, 22--26 June 1993, pp. 148-- 155.

....is placed at the corresponding location on the x axis. These data are derived from the dotplot shown in figure 1. Actual boundaries correspond to the most extreme minima those at positions 1085, 2206 and 2863. Results Since determining where topic boundaries belong is a subjective task, (Passoneau and Litman, 1993), the preliminary experiments conducted using this algorithm involved discovering boundaries between concatenated articles. All of the articles were from the Wall Street Journal and were tagged in conjunction with the Penn Treebank project, which is described in (Marcus et al. 1993) The ....

Passoneau, Rebecca J. and Diane J. Litman. IntentionBased Segmentation: Human Reliability and Correlation with Linguistic Cues. Proceedings of the 31st Annual Meeting of the Association for Computational Linguistics, 1993.

....where the models used are similar in form to the models described here [25] There has been recent growth in empirically oriented work in discourse processing. Several researchers have addressed evaluation, investigating the degree to which human subjects agree with one another on discourse tasks [52], 35] 33] Others have used frequency information to evaluate algorithms. Passonneau and Litman [52] tagged a corpus with classes and features and tested algorithms hypothesized from the literature. They consider just one feature at a time, so do not address interactions among multiple ....

....in empirically oriented work in discourse processing. Several researchers have addressed evaluation, investigating the degree to which human subjects agree with one another on discourse tasks [52] 35] 33] Others have used frequency information to evaluate algorithms. Passonneau and Litman [52] tagged a corpus with classes and features and tested algorithms hypothesized from the literature. They consider just one feature at a time, so do not address interactions among multiple features. Some researchers have derived algorithms and models on the basis of frequency information, but have ....

Passonneau, R. and Litman, D. (1993). Intention-Based Segmentation: Human Reliability and Correlation with Linguistic Cues. Procs. 31st Annual Meeting of the Assoc. for Computational Linguistics (ACL-93), pp. 148-155.

.... Nakatani, 1996) we performed an intercoder reliability study investigating agreement in annotating the times. The main goal in developing annotation instructions is to make them precise but intuitive so that they can be used reliably by nonexperts after a reasonable amount of training (see Passonneau Litman, 1993; Condon Cech, 1995; Hirschberg Nakatani, 1996) Reliability is measured in terms of the amount of agreement among annotators; high reliability indicates that the encoding scheme is reproducible given multiple annotators. In addition, the instructions also serve to document the annotations. ....

Passonneau, R., & Litman, D. (1993). Intention-based segmentation: human reliability and correlation with linguistic cues. In Proc. of the 31st Annual Meeting of the Association for Computational Linguistics, pp. 148--155.

....Each of the 20 articles in the corpus was segmented by at least four human judges, and the majority opinion of segment boundaries was computed as the evaluation standard (Klavans et al. 1998) Human judges achieved on average only 62.4 agreement with the majority opinion, as seen in Table 2. Passonneau and Litman (1993) show that this surprisingly low agreement is often the result of evaluators being divided between those who regard segments as more localized and those who prefer to split only on large boundaries. We then verified that the task was well defined by testing for a strong correlation between the ....

....segments as more localized and those who prefer to split only on large boundaries. We then verified that the task was well defined by testing for a strong correlation between the markings of the human judges. We test for inter judge reliability using Cochran (1950) s Qtest, also discussed in Passonneau and Litman (1993). We found a very high correlation between judges indicating that modeling the task was indeed feasible; the results showed that there was less than a 0.15 chance on average that the judges segment marks agreed by chance. We also calculated Kappa (K) another correlation statistic that corrects ....

Passonneau R. J. and Litman D. J. (1993) Intentionbased segmentation: human reliability and correlation with linguistic cues. Proceeding of the 31st Annual Meeting of the Association of Computation Linguistics, pp. 148-155.

....Kowtko et al. 5] having devised a coding scheme for conversational moves involving thirteen distinctions, cite separate pairwise agreement percentages between one expert coder and each of three naive coders in order to argue that the coding scheme is replicable. Meanwhile, Passonneau and Litman [7], in arguing that naive subjects can reliably agree on whether or not given prosodic phrase boundaries are also discourse segment boundaries, measure replicability using percent agreement , defined (in [4] as the ratio of observed agreements with the majority opinion to possible agreements with ....

R. J. Passonneau and D. J. Litman. Intention-based segmentation: human reliability and correlation with linguistic cues. In Proceedings of the 31st Annual Meeting of the ACL, pages 148--155, June 1993.

No context found.

. Association for Computational Linguistics. 148--155.

....for writing sentences (1.3 5) perhaps to clarify for the reader what was written in (1.2) The details given in (1.3 5) serve this purpose of the author. There is evidence that people do perform this kind of segmentation of discourse during understanding. For example, Passonneau and Litman [19] and Hirschberg and Grosz [7] found 1 Eric S. Roberts, The Art and Science of C: A Library Based Introduction to Computer Science, AddisonWesley, 1995, page xv. 3 statistically significant agreement among subjects who were asked to perform a discoursesegmentation task. How might a computer ....

Passonneau, Rebecca and Litman, Diane. Intention-based segmentation: human reliability and correlation with linguistic cues. In Proceedings of the 31st meeting of the 16 Association for Computational Linguistics, (June 22--26, Columbus, Ohio). Association for Computational Linguistics, N.J., 1993, pp. 148--155.

No context found.

Rebecca J. Passonneau and Diane J. Litman. Intention-Based Segmentation: Human Reliability and Correlation with Linguistic Cues. In Proceedings of the 31st Annual Meeting of the ACL, pages 148--155, Columbus, Ohio, June 1993. Association for Computational Linguistics.

No context found.

Passonneau, R., Litman, D.J., `Intention-based Segmentation: Human Reliability and Correlation with Linguistic Cues', Proceedings of the 31th Annual Meeting, pp. 148-155, ACL, 1993.

No context found.

R. Passonneau, D.J. Litman, Intention-based segmentation: Human reliability and correlation with linguistic cues, Proceeding of the 31th Annual Meeting, 148-155, ACL.

No context found.

Passonneau, R.J. & Litman, D.J. (1993). Intention-based segmentation: human reliability and correlation with linguistic cues. In Proc. of the 31st Annual Meeting of the ACL, pp. 148-155.

No context found.

Rebecca J. Passonneau and Diane J. Litman. 1993. Intention-based segmentation: Human reliability and correlation with linguistic cues. In ACL 31, pages 148-155.

No context found.

Kyoto. Rebecca J. Passonneau and Diane J. Litman. 1993. "Intention-based Segmentation: Human Reliability and Correlation with Linguistic Cues" Proceedings of the 31st Annual Meeting of the Association for Computational Linguistics. ACL.

No context found.

Kyoto. Rebecca J. Passonneau and Diane J. Litman. 1993. "Intention-based Segmentation: Human Reliability and Correlation with Linguistic Cues" Proceedings of the 31st Annual Meeting of the Association for Computational Linguistics. ACL.

No context found.

Passonneau, R. & Litman, D. (1993). Intention-Based Segmentation: Human Reliability and Correlation with Linguistic Cues. In Proc. of ACL-93.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC