Finally we come to the last week of the 2 month Trainer MOOC. And we are covering a very interesting topic and that is measuring training effectiveness.
It is interesting because it is also time for us to evaluate our training program…Now since a MOOC is so different than traditional training the question arises “How to evaluate whether it was a success?” Some idea’s on metrics that may be worthwhile to look at:
- No of registered learners
- Discussion generated : Quality and quantity of comments posted
- Pre and post studies of:
- Increase in traffic – number of people who visit the blog and especially the number of repeat visitors
- Number of page views i.e when a person visits the blog how many pages does he/she view
- Average time spent on the blog
- How many emails received by the learners
- Increase in twitter follows (we had asked people to follow on twitter)
- How many mentions/ interactions on twitter
Additionally while we had free learners who were learning online we had paid participants who were much more involved since they had to ensure they had evidence of learning at the training clinic.
A key aspect is that before commencing the MOOC and training clinic we needed to decide how will we evaluate the program.
Since we have attended training on the ROI methodology by Patti Phillip’s we have a bias towards it, so we look at the 5 levels of evaluation (Donald Kirkpatrick, 1959 Levels 1 – 4; Jack Phillips, 1970 Level 5) to evaluate the MOOC:
- Level 1: Reactions (the “smiley” sheet)
- Did participants like the training they received?
- Level 2: Learning
- Are participants confident that they have learned something from the training program?
- Level 3: Behavior/Application
- Are participants able to apply what they learned in the training program back on the job?
- Level 4: Results/Business Impact
- Did the training show improvement in efficiencies, productivity, profits, costs, reduced turnover?
- Level 5: ROI
- Did the training program show a positive ROI?
For the paid participants where we had the training clinic we were at Level 3. We ensured that the learners took the step of applying what they learnt on the job.
Now for the free participants…hmm….I really can’t assess….as the learners have mostly taken a passive role….reading yet not commenting or sharing their reflections. Most of the interactions for queries they had or reflections that they made especially on key issues raised was done by email and not via comments or their own blogs 
It raises the question that can the ROI methodology actually evaluate a MOOC? Pl share your views…
Also as new methodologies of training are created the need of new models of evaluation will also become evident.
Resources on evaluating training:
- Kirkpatrick’s model: http://www.businessballs.com/kirkpatricklearningevaluationmodel.htm
- Phillip’s ROI methodology http://leanlearning.wikispaces.com/learning_analytics
- CIRO(Context, Input, Reaction, Outcome) and Kaufman’s Five Levels of Evaluation www.ttefdergi.gazi.edu.tr/makaleler/2005/Sayi1/155-169.pdf
Evaluation while improves the learning process by bringing accountability also costs money…that is a deterrent for organizations.
Happy evaluating!!!
Facilitators:
- Lovely Kumar – Chief Projects
- Anila Rattan – Chief Strategy
Larks Learning
Contact:lovely@larkslearning.com
To know more about our training and assessment services please contact: lovely@larkslearning.com