Results 1 - 10
of
15
Methods for ordinal peer grading
- In KDD, KDD ’14, ACM
, 2014
"... Massive Online Open Courses have become an accessible and affordable choice for education. This has led to new technical challenges for instructors such as student evaluation at scale. Recent work has found ordinal peer grading, where individ-ual grader orderings are aggregated into an overall order ..."
Abstract
-
Cited by 4 (1 self)
- Add to MetaCart
(Show Context)
Massive Online Open Courses have become an accessible and affordable choice for education. This has led to new technical challenges for instructors such as student evaluation at scale. Recent work has found ordinal peer grading, where individ-ual grader orderings are aggregated into an overall ordering of assignments, to be a viable alternate to traditional instruc-tor/staff evaluation [23]. Existing techniques, which extend rank-aggregation methods, produce a single ordering as out-put. While these rankings have been found to be an accurate reflection of assignment quality on average, they do not com-municate any of the uncertainty inherent in the assessment process. In particular, they do not to provide instructors with an estimate of the uncertainty of each assignment’s position in the ranking. In this work, we tackle this problem by ap-plying Bayesian techniques to the ordinal peer grading prob-lem, using MCMC-based sampling techniques in conjunction with the Mallows model. Experiments are performed on real-world peer grading datasets, which demonstrate that the pro-posed method provides accurate uncertainty information via the estimated posterior distributions.
Autonomously generating hints by inferring problem solving policies
- In Proceedings of the Second (2015) ACM Conference on Learning@ Scale, L@S ’15
, 2015
"... Exploring the whole sequence of steps a student takes to pro-duce work, and the patterns that emerge from thousands of such sequences is fertile ground for a richer understanding of learning. In this paper we autonomously generate hints for the Code.org ‘Hour of Code, ’ (which is to the best of our ..."
Abstract
-
Cited by 3 (3 self)
- Add to MetaCart
(Show Context)
Exploring the whole sequence of steps a student takes to pro-duce work, and the patterns that emerge from thousands of such sequences is fertile ground for a richer understanding of learning. In this paper we autonomously generate hints for the Code.org ‘Hour of Code, ’ (which is to the best of our knowledge the largest online course to date) using historical student data. We first develop a family of algorithms that can predict the way an expert teacher would encourage a student to make forward progress. Such predictions can form the ba-sis for effective hint generation systems. The algorithms are more accurate than current state-of-the-art methods at recreat-ing expert suggestions, are easy to implement and scale well. We then show that the same framework which motivated the hint generating algorithms suggests a sequence-based statis-tic that can be measured for each learner. We discover that this statistic is highly predictive of a student’s future success.
Scaling Short-answer Grading by Combining Peer Assessment with Algorithmic Scoring
"... Peer assessment helps students reflect and exposes them to different ideas. It scales assessment and allows large online classes to use open-ended assignments. However, it requires students to spend significant time grading. How can we lower this grading burden while maintaining quality? This paper ..."
Abstract
-
Cited by 3 (0 self)
- Add to MetaCart
(Show Context)
Peer assessment helps students reflect and exposes them to different ideas. It scales assessment and allows large online classes to use open-ended assignments. However, it requires students to spend significant time grading. How can we lower this grading burden while maintaining quality? This paper integrates peer and machine grading to preserve the robust-ness of peer assessment and lower grading burden. In the identify-verify pattern, a grading algorithm first predicts a student grade and estimates confidence, which is used to es-timate the number of peer raters required. Peers then iden-tify key features of the answer using a rubric. Finally, other peers verify whether these feature labels were accurately ap-plied. This pattern adjusts the number of peers that evaluate an answer based on algorithmic confidence and peer agree-ment. We evaluated this pattern with 1370 students in a large, online design class. With only 54 % of the student grading time, the identify-verify pattern yields 80-90 % of the accu-racy obtained by taking the median of three peer scores, and provides more detailed feedback. A second experiment found that verification dramatically improves accuracy with more raters, with a 20 % gain over the peer-median with four raters. However, verification also leads to lower initial trust in the grading system. The identify-verify pattern provides an ex-ample of how peer work and machine learning can combine to improve the learning experience. Author Keywords assessment; online learning; automated assessment; peer learning
Strategyproof peer selection: Mechanisms, analyses, and experiments
- In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI
, 2016
"... We study an important crowdsourcing setting where agents evaluate one another and, based on these evaluations, a sub-set of agents are selected. This setting is ubiquitous when peer review is used for distributing awards in a team, allocat-ing funding to scientists, and selecting publications for co ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
We study an important crowdsourcing setting where agents evaluate one another and, based on these evaluations, a sub-set of agents are selected. This setting is ubiquitous when peer review is used for distributing awards in a team, allocat-ing funding to scientists, and selecting publications for con-ferences. The fundamental challenge when applying crowd-sourcing in these settings is that agents may misreport their reviews of others to increase their chances of being se-lected. We propose a new strategyproof (impartial) mecha-nism called Dollar Partition that satisfies desirable axiomatic properties. We then show, using a detailed experiment with parameter values derived from target real world domains, that our mechanism performs better on average, and in the worst case, than other strategyproof mechanisms in the literature. 1
Creating Awareness and Reflection in a Large-Scale IS Lecture – The Application of a Peer Assessment in a Flipped Classroom Scenario
"... Abstract. Large-scale lectures are a typical way of teaching university students. However, these lectures often lack interaction elements and do not foster awareness and reflection in the learning process. This results in insufficient learning outcomes such as learning satisfaction and success. Ther ..."
Abstract
-
Cited by 1 (1 self)
- Add to MetaCart
(Show Context)
Abstract. Large-scale lectures are a typical way of teaching university students. However, these lectures often lack interaction elements and do not foster awareness and reflection in the learning process. This results in insufficient learning outcomes such as learning satisfaction and success. Therefore, a new approach to engage interaction in such large-scale lectures is the flipped class-room concept which seeks to overcome these challenges by stimulating self-regulated learning phases and improving interaction as well as awareness and reflection in the presence phases of a lecture. However, it is still unclear how to actually increase reflection and awareness through interaction in such learning scenarios. For this purpose, we propose an application of a technology-enhanced peer assessment that is carried out in large-scale information systems lectures. Preliminary evaluation results suggest the potentials of this approach. Thus, we are able to provide first theoretical and practical implications for the application of a technology-enhanced peer assessment in large-scale lectures.
Structuring Interactions for Large-Scale Synchronous Peer Learning
"... This research investigates how to introduce synchronous in-teractive peer learning into an online setting appropriate both for crowdworkers (learning new tasks) and students in mas-sive online courses (learning course material). We present an interaction framework in which groups of learners are for ..."
Abstract
- Add to MetaCart
(Show Context)
This research investigates how to introduce synchronous in-teractive peer learning into an online setting appropriate both for crowdworkers (learning new tasks) and students in mas-sive online courses (learning course material). We present an interaction framework in which groups of learners are formed on demand and then proceed through a sequence of activi-ties that include synchronous group discussion about learner-generated responses. Via controlled experiments with crowd-workers, we show that discussing challenging problems leads to better outcomes than working individually, and incentiviz-ing people to help one another yields still better results. We then show that providing a mini-lesson in which workers con-sider the principles underlying the tested concept and justify their answers leads to further improvements. Combining the mini-lesson with the discussion of the multiple-choice ques-tion leads to significant improvements on that question. We also find positive subjective responses to the peer interactions, suggesting that discussions can improve morale in remote work or learning settings.
Mudslide: A Spatially Anchored Census of Student Confusion for Online Lecture Videos
"... Educators have developed an effective technique to get feedback after in-person lectures, called “muddy cards.” Students are given time to reflect and write the “muddiest” (least clear) point on an index card, to hand in as they leave class. This practice of assigning end-of-lecture reflection tasks ..."
Abstract
- Add to MetaCart
(Show Context)
Educators have developed an effective technique to get feedback after in-person lectures, called “muddy cards.” Students are given time to reflect and write the “muddiest” (least clear) point on an index card, to hand in as they leave class. This practice of assigning end-of-lecture reflection tasks to generate explicit student feedback is well suited for adaptation to the challenge of supporting feedback in online video lectures. We describe the design and evaluation of Mudslide, a prototype system that translates the practice of muddy cards into the realm of online lecture videos. Based on an in-lab study of students and teachers, we find that spatially contextualizing students ’ muddy point feedback with respect to particular lecture slides is advantageous to both students and teachers. We also reflect on further opportunities for enhancing this feedback method based on teachers ’ and students ’ experiences with our prototype. Author Keywords Flipped classrooms; lecture feedback; online lecture videos
Incorporating Collaborative Learning Into xMOOCs: A Proposal of Future Research Directions
"... Facilitating collaborative learning in Massive Online Open Course (MOOC) has been a key research interest in recent years. Studies so far have shown promising signs of improving learners ’ retention and engagement in MOOCs by incorporating collaborative learning and social interaction onto the platf ..."
Abstract
- Add to MetaCart
(Show Context)
Facilitating collaborative learning in Massive Online Open Course (MOOC) has been a key research interest in recent years. Studies so far have shown promising signs of improving learners ’ retention and engagement in MOOCs by incorporating collaborative learning and social interaction onto the platform. While a few works have found improvements in certain aspects of learning with existing collaborative features in MOOCs, further investigation is needed on how they should be designed and incorporated. In this paper, we provide a brief literature review of peer learning on MOOCs, propose research gaps in this area, suggest future directions on this topic based on previous literature and reports, and conclude by outlining our focuses and plans in this research area. Author Keywords Collaborative learning, MOOCs, design considerations
Uncovering Trajectories of Informal Learning in Large Online Communities Of Creators
"... We analyzed informal learning in Scratch Online – an online community with over 4.3 million users and 6.7 million instances of user-generated content. Users develop projects, which are graphical interfaces consisting of interacting programming blocks. We investigated two fundamental questions of how ..."
Abstract
- Add to MetaCart
(Show Context)
We analyzed informal learning in Scratch Online – an online community with over 4.3 million users and 6.7 million instances of user-generated content. Users develop projects, which are graphical interfaces consisting of interacting programming blocks. We investigated two fundamental questions of how we can model informal learning, and which patterns of informal learning emerge. We proceeded in two phases. First, we modeled learning as a trajectory of cumulative programming block usage by long-term users who created at least 50 projects. Second, we applied K-means++ clustering to uncover patterns of learning and corresponding subpopulations. We found four groups of users manifesting four different patterns of learning, ranging from the smallest to the largest improvement. At one end of the spectrum, users learned more and in a faster manner. At the opposite end, users did not show much learning progress, even after creating dozens of projects. The modeling and clustering of trajectory patterns that enabled us to quantitatively analyze informal learning may be applicable to other similar communities. The results can also support administrators of online communities in implementing customized interventions for specific subpopulations. Author Keywords Learning analytics; informal learning; modeling; clustering;
Structuring Interactions for Large-Scale Synchronous Peer Learning
"... This research investigates how to introduce synchronous in-teractive peer learning into an online setting appropriate both for crowdworkers (learning new tasks) and students in mas-sive online courses (learning course material). We present an interaction framework in which groups of learners are for ..."
Abstract
- Add to MetaCart
(Show Context)
This research investigates how to introduce synchronous in-teractive peer learning into an online setting appropriate both for crowdworkers (learning new tasks) and students in mas-sive online courses (learning course material). We present an interaction framework in which groups of learners are formed on demand and then proceed through a sequence of activi-ties that include synchronous group discussion about learner-generated responses. Via controlled experiments with crowd-workers, we show that discussing challenging problems leads to better outcomes than working individually, and incentiviz-ing people to help one another yields still better results. We then show that providing a mini-lesson in which workers con-sider the principles underlying the tested concept and justify their answers leads to further improvements. Combining the mini-lesson with the discussion of the multiple-choice ques-tion leads to significant improvements on that question. We also find positive subjective responses to the peer interactions, suggesting that discussions can improve morale in remote work or learning settings.
