Results 1 
8 of
8
Automated feedback generation for introductory programming assignments
, 2013
"... We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, th ..."
Abstract

Cited by 25 (7 self)
 Add to MetaCart
(Show Context)
We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal corrections to student’s incorrect solutions, providing them with a measure of exactly how incorrect a given solution was, as well as feedback about what they did wrong. We introduce a simple language for describing error models in terms of correction rules, and formally define a ruledirected translation strategy that reduces the problem of finding minimal corrections in an incorrect program to the problem of synthesizing a correct program from a sketch. We have evaluated our system on thousands of real student attempts obtained from the Introduction to
Synthesis of Geometry Proof Problems
"... This paper presents a semiautomated methodology for generating geometric proof problems of the kind found in a highschool curriculum. We formalize the notion of a geometry proof problem and describe an algorithm for generating such problems over a userprovided figure. Our experimental results i ..."
Abstract

Cited by 1 (1 self)
 Add to MetaCart
This paper presents a semiautomated methodology for generating geometric proof problems of the kind found in a highschool curriculum. We formalize the notion of a geometry proof problem and describe an algorithm for generating such problems over a userprovided figure. Our experimental results indicate that our problem generation algorithm can effectively generate proof problems in elementary geometry. On a corpus of 110 figures taken from popular geometry textbooks, our system generated an average of about 443 problems per figure in an average time of 4.7 seconds per figure. 1
Automatic Problem Generation for CapturetheFlag Competitions
"... Abstract Computer security games, especially capturetheflag (CTF) competitions, are growing in popularity. A typical CTF contest presents users with a set of hacking challenges, where correct solutions reveal a text "flag" that can be submitted to a scoring server. In traditional CTF ar ..."
Abstract
 Add to MetaCart
(Show Context)
Abstract Computer security games, especially capturetheflag (CTF) competitions, are growing in popularity. A typical CTF contest presents users with a set of hacking challenges, where correct solutions reveal a text "flag" that can be submitted to a scoring server. In traditional CTF architectures, the problem and the flag are the same across the competition. In this paper we discuss automatic problem generation (APG), where a given challenge is not fixed, but rather can have many different automatically generated problem instances. APG offers players a unique competition experience and can facilitate deliberate practice where problems vary just enough to make sure a user can replicate the solution idea. APG also allows competition administrators the ability to detect when users submit a copied flag from another user to the scoring server. In 2014 we ran a largescale CTF competition called PicoCTF, where we measured the prevalence of flag sharing. Our results indicate that about 0.8% of flags submitted to AGP problems were copied, with 14% of teams submitting at least one shared flag. In 68% of flag sharing cases, teams went on to eventually solve the problem on their own.
Formal Methods for ComputerAided STEM Education
"... Providing personalized and interactive education (as in oneonone tutoring) remains an unsolved problem for standard classrooms. The arrival of Massive Open Online Courses (MOOCs), while having provided a unique opportunity to share quality instruction with massive number of students, only exacerba ..."
Abstract
 Add to MetaCart
(Show Context)
Providing personalized and interactive education (as in oneonone tutoring) remains an unsolved problem for standard classrooms. The arrival of Massive Open Online Courses (MOOCs), while having provided a unique opportunity to share quality instruction with massive number of students, only exacerbates this problem with an even higher student to teacher ratio. We believe that automated intelligent tutoring can play a revolutionary role in this context. In this article, we motivate four problem definitions, namely problem generation, solution generation, feedback generation, and content authoring in the context of intelligent tutoring. We describe how formal methods can play a useful role in addressing these problems. We present some recent results that have been applied to a variety of STEM subject domains (including logic, automata, programming, arithmetic, algebra, and geometry) as illustrative examples.
9How Can Automatic Feedback Help Students Construct Automata?
"... In computeraided education, the goal of automatic feedback is to provide a meaningful explanation of students ’ mistakes. We focus on providing feedback for constructing a deterministic finite automaton that accepts strings that match a described pattern. Natural choices for feedback are binary fe ..."
Abstract
 Add to MetaCart
(Show Context)
In computeraided education, the goal of automatic feedback is to provide a meaningful explanation of students ’ mistakes. We focus on providing feedback for constructing a deterministic finite automaton that accepts strings that match a described pattern. Natural choices for feedback are binary feedback (correct/wrong) and a counterexample of a string that is processed incorrectly. Such feedback is easy to compute but might not provide the student enough help. Our first contribution is a novel way to automatically compute alternative conceptual hints. Our second contribution is a rigorous evaluation of feedback with 377 students. We find that providing either counterexamples or hints is judged as helpful, increases student perseverance, and can improve problem completion time. However, both strategies have particular strengths and weaknesses. Since our feedback is completely automatic, it can be deployed at scale and integrated into existing massive open online courses.
Automatic Generation of Alternative Starting Positions for Simple Traditional Board Games
"... Simple board games, like TicTacToe and CONNECT4, play an important role not only in the development of mathematical and logical skills, but also in the emotional and social development. In this paper, we address the problem of generating targeted starting positions for such games. This can faci ..."
Abstract
 Add to MetaCart
Simple board games, like TicTacToe and CONNECT4, play an important role not only in the development of mathematical and logical skills, but also in the emotional and social development. In this paper, we address the problem of generating targeted starting positions for such games. This can facilitate new approaches for bringing novice players to mastery, and also leads to discovery of interesting game variants. We present an approach that generates starting states of varying hardness levels for player 1 in a twoplayer board game, given rules of the board game, the desired number of steps required for player 1 to win, and the expertise levels of the two players. Our approach leverages symbolic methods and iterative simulation to efficiently search the extremely large state space. We present experimental results that include discovery of states of varying hardness levels for several simple gridbased board games. The presence of such states for standard game variants like 4 × 4 TicTacToe opens up new games to be played that have never been played as the default start state is heavily biased. 1