Forum Discussion
Scaling feedback?
Do you have any way of providing feedback to your learners? If so, how do you think about scaling it?
Hey all! I recently had a coffee chat with JulieDirksen and she said something that got me thinking (well, she said a lot of things that got me thinking but this was one I wanted to share!).
She was talking about how for a long time, feedback was one of the last things in learning that you couldn't scale. But, she noted that with AI that is now changing—it's now more possible to scale the experience of receiving feedback on your work.
As a community professional, scaling feedback is something I've long thought about doing via nominating experts from the community to help spread the expertise of an "instructor" beyond one single person—I'm actually going to be giving a session at Articuland next week in Orlando that touches on this!
But, all of this got me thinking about different forms of providing feedback and how they have the potential to scale, and I wanted to pitch this question to the community and get your thoughts.
9 Replies
Julie's right that with AI something's changed, but I think the hard part is now task design rather than the feedback itself. Most Rise and SL courses give the learner nothing to get feedback on. A tick or a cross tells you nothing about what they were thinking. Ask them to put the steps of a process in order, or to spot what's wrong in a worked example, and now you can see where they went wrong and say so. None of that needs AI either, it's just triggers and conditions.
Open response is where the AI could come in, and I'm a bit torn on it. I built a weekly challenge course that highlighted this, The Red Pen, and came away less convinced than I went in. Give a model a generic rubric and it produces confident, even-handed comments on everything, including the bits that don't matter. Learners trust that. At scale I suspect it's worse than no feedback at all, because nobody's questioning it…
Really interesting point. I wonder whether this is really just where community comes in. I suspect peer feedback is better than AI feedback. I think about something like the weekly challenges here that DavidAnderson runs and how those make such a difference to learning in comparison to just taking async training. I think the social learning and feedback is an aspect of that.
Maybe the honest answer is that it doesn't scale… I guess this is why using AI is tempting, and why I'm still wary of it!
I think the type of feedback matters a lot when deciding whether it can actually scale. For objective tasks, automated feedback can work well, but for open-ended work, the value often comes from understanding the learner's reasoning rather than just identifying whether the answer is correct.
AI could help with the first layer of feedback, such as pointing out patterns or suggesting areas to revisit, while peer or instructor feedback could handle the parts that require more context and judgment. That might make scaling more realistic without relying on AI to replace the human side completely.
How do others decide which parts of learner feedback are suitable for automation and which should remain human-led?
I really like this answer alextaylorone — I like the idea of being critical about which parts of the feedback motion are possible to automate and which require a human in the loop. It allows us to not be so black and white about saying "AI can't replace human feedback" so that a piece of the motion can scale with the help of technology, while still ensuring quality.
Have we connected in the community before? I like how you think! It would be great to have a virtual coffee chat some time :)
Scaling feedback is such a succinct way to phrase the problem. All effective learning seems to require some kind of feedback, so how can we add more as well as multiply the impact of what's there?
Scaling could be along channels of information. We can expect an audio-less black-and-white educational video to have missed some opportunities to reinforce key ideas compared to a video that consciously uses color to show relationships between objects on screen and uses audio to add emotional impact.
To me lately, scaling feedback has meant a scripted system of procedurally generated feedback. The answers chosen in an interactive eLearning could be tracked by holistic variables like "amountOfRiskTaken" or "customerSatisfaction," and then the quiz can present conditional feedback to the learner according to where they fell on that measure. For learners who chose more answers that were technically correct but also resulted in a negative customer satisfaction, the eLearning could provide additional feedback reminding them of the customer. While this kind of functionality doesn't help scale access to the content, it does scale the effectiveness of the content by giving some of the most immediate feedback the learner can receive.
Love these solutions to provide feedback in the flow of a course just simply based on the course design, not based on a human providing in-the-moment feedback! Thanks for your reply!
This maps to what I've seen on the AI-integration side: feedback scales well when the response space is constrained. Branched scenarios and knowledge checks with pre-defined paths? Scripted or AI-assisted feedback works beautifully there, because the rubric is baked into the design. Open response is where it gets shaky, and DShaw's rubric point nails why — hand a model a generic rubric and you get confident-sounding mush. The AI feedback I've found actually useful sat on top of rubrics written like a strict SME wrote them: specific examples of what good looks like, not adjectives.
The other decision I'd force early: where does the human sit in the loop? AI as a first-pass draft for a human reviewer to approve is a very different scaling story than AI as the final voice to the learner. In regulated training especially, that second one needs a paper trail and a person accountable for it. My default is AI-drafts-human-signs-off, and fully automated feedback only for low-stakes, objective checks.
Ooooo this is gold ronald-hicks : "specific examples of what good looks like, not adjectives." Thank you for chiming in!
