In a proof-of-concept experiment on using natural language processing (NLP) to provide effective student feedback, 949 participants from the UK (paid \$1 each) completed a learning activity on identifying poison ivy. Participants first received one of two randomly assigned explanations to induce prior knowledge. They then took a six-question multiple-choice pre-test. Afterward, they answered four similar questions requiring a short written explanation of their reasoning. Based on these written responses, participants received one of two targeted feedback paragraphs. Finally, participants completed a post-test identical to the pre-test but with randomized question order. Researchers evaluated the NLP feedback's effectiveness by measuring learning gains (post-test minus pre-test scores).

Adaptive Natural-Language Targeting for Student Feedback: Experimental Procedure

The method used here targets feedback, not by semantic similarity, but by predicted results based on the end goal of knowledge gain. This requires the hand-creation of a set of feedback texts useful for the task at hand, and this reduces the labor cost relative to other hand-tailored methods.

University of Georgia

The goal of this study is to work toward developing more effective feedback-targeting systems for tutoring software, to improve user performance. In this study, researchers built a tutoring software system that uses Natural Language Processing (NLP) to provide adaptive feedback to students in an online learning task. 
- The researcher's system models the interaction between a student's natural language response and the software, which interprets students' responses and provides the best feedback action. This framework allows for the optimization of a reward signal. 
- Researchers showed that natural-language based targeting policies were able to choose optimal feedback from fewer exercises compared to multiple-choice based policies. The natural-language policies were effective even when tested on novel interactions. 

Adaptive Natural-Language Targeting for Student Feedback- Goal of Study

Kolchinski, Y. A., Ruan, S., Schwartz, D., & Brunskill, E. (2018). Adaptive natural-language targeting for student feedback. Proceedings of the Fifth Annual ACM Conference on Learning at Scale. doi:10.1145/3231644.3231684


Adaptive Natural-Language Targeting for Student Feedback

Natural language processing- The interaction between computers and humans via natural language
Reward signal- Pre-test to post-test gain, based on the scores
Semantic matching- A technique used in computer science to identify information that is semantically related.

Adaptive Natural-Language Targeting for Student Feedback- Relevant Terms

An important difference between human and software tutors is their ability to use natural language. Human tutors are effective in explaining concepts and gauging what students understand, but software is usually only effective in presenting natural-language material to students, and not in interpreting the natural-language responses of a student. It was found that using the natural language processing (NLP) targeted strategy was more effective in providing optimal feedback compared to traditional means of feedback targeting, like multiple-choice targeting. 

Adaptive Natural-Language Targeting for Student Feedback- Background

Previous tutoring systems have given feedback in several contexts, including correcting mistakes, providing hints, and metacognitive feedback meant to encourage good habits. The predominant ways in which previous systems targeted their feedback were based on task-specific inputs such as multiple-choice responses, computer code, and math equations written by the student. These systems are missing out because they use less expressive inputs for targeting feedback to students. Other systems have implemented the use of NLP for providing feedback, but these methods proved to have limitations of feedback targeting mechanisms, which depend on a combination of hand-tuned knowledge engineering and semantic matching. These methods are extremely effort-intensive and are not feasible on a larger scale.

Adaptive Natural-Language Targeting for Student Feedback- Previous Techniques

The researchers sought to evaluate other strategies of feedback-targeting, using the collected data, to evaluate the performance of the researcher's NLP strategy, which was used to pick out the right feedback message from the response. Researchers implemented and tested four strategies to evaluate in regards to learning gain: oracle, multiple-choice targeted policy, and two NLP targeted policies. There was a 95% confidence interval used in these tests. Participants' learning gains by four different one-questions policies were compared and results showed that policies targeted through NLP outperformed multiple-choice targeted strategies on all of the exercises except the last one (3/4 exercises). This shows the effectiveness of using NLP responses, rather than multiple-choice responses to provide the most effective feedback. Furthermore, compared to multiple-choice responses alone, the NLP strategy proved to contain more variation between questions. Unlike the multiple-choice models, the NLP models can generalize to unseen questions by learning a general notion of how students respond to prompts depending on what they know. These results show the promising effects of using natural-language inputs from students for feedback selection.


Learn Before

Related

Learn After