By Imon Ghosh
Abstract
Since its introduction in 1959, the Kirkpatrick Model for evaluating training has gained wide acceptance in a variety of organizations. Has this venerable model outlived its utility? This paper outlines the Kirkpatrick Model, discusses some of its shortcomings, and explores the models for training evaluation that may eventually replace the Kirkpatrick Model, including the T.V. Rao Learning Systems Framework, the Rothwell et al Model, the Success Case Method, Goleman’s “Optimal Process” model, and the American Heart Association Training Evaluation Model that specifies Client Outcomes and Community Impacts, in addition to Organisational Impact.
Introduction
Donald L Kirkpatrick first published his ideas on training evaluation in 1959, in a series of articles in the US Training and Development Journal. The articles were subsequently included in Kirkpatrick's book Evaluating Training Programs (1975), published by the American Society for Training and Development (ASTD). The Kirkpatrick model of training evaluation proposed evaluating training on four levels:
Level 1 Evaluation: 1st Reaction
Level 2 Evaluation: Learning transfer
Level 3 Evaluation: Behavioral change
Level 4 Evaluation: Positive, attributable results
Level 1 - 1st Reaction
The first and most immediate level aims to measure participants’ initial reactions to the training program. There is debate about the usefulness of this measure especially regarding the superficial nature of simple evaluation documents.
A common issue associated with level one evaluation is the superficial nature of the information gathered. This does not have to be the case. The quality of the questions determines the usefulness of the information that can be gathered.
Learning (Level two outcomes) and transfer of learning (Level three outcomes) is more likely to occur when participants have positive attitudes toward the training program. Those measuring training effectiveness need to establish performance criteria against results in the workplace, so that common evaluative judgments can be made on all four levels of Kirkpatrick’s outcome evaluation.
Level 2 - Learning Transfer
Level two evaluation is done soon after the training event to determine if participants gained the knowledge, skills, or attitudes.
Where training is effective and provides delegates with useful guidance they will almost certainly make use of knowledge reviews, notice their own progress and be able to recall specific instances of improvement.
Level 3 - Behavioral Changes
At Level 3, the Kirkpatrick goal is to find out if training program participants change their on the job behavior as a result of their attendance and participation in the training program. Learning is likely to transfer more effectively if the conditions in the work setting are favorable and there is a widespread demand for such transfer.
Level 4 - Positive and Attributable Results
The final level aims to find out if the training program led to final business results that contribute to the “bottom line” (i.e., business profits). Level four outcomes are not limited to return on training investment; they can include other major results that contribute to the effective functioning of an organization. These include outcomes that most people would agree is “good for the business.” I have described these business impacts in an article in Newstalk, vol 12 no. 3 published in March 2004, on The Benefits of Training:
How training benefits the organization:
• Improves the job knowledge and skill at all levels of the organization.
• Improves individual, team and corporate performance in terms of output, quality, speed and overall productivity.
• Improves operational flexibility by extending the range of skills possessed by employees (multi-skilling).
• Improves the morale of the work force.
• Attracts high-quality employees by offering them learning and development opportunities.
• Increases levels of competence and enhances skills, thereby enabling employees to obtain more job satisfaction, gain higher rewards and progress within the organization.
• Increases the commitment of employees by encouraging them to identify with the mission and objectives of the organization.
• Helps create a better corporate image.
• Aids in organizational development.
• Aids in understanding and carrying out organizational policies.
• Organization gets more effective decision making and problem solving.
• Aids in development for promotion from within.
• Aids in developing leadership skill, motivation, loyalty, better attitudes, and other aspects that successful workers and managers usually display.
• Aids in increasing productivity and/or quality of work.
• Helps keep costs down in many areas, e.g. production, personnel, administration, etc.
• Stimulates preventive management as opposed to putting out fires.
• Creates an appropriate climate for growth.
• Aids in improving organizational communication.
• Aids in handling conflict, thereby helping to prevent stress and tension.
• Helps to manage change by increasing understanding of the reasons for change, and providing employees with the knowledge and skills they need to adjust to new situations.
• Helps to develop a positive culture in the organization, which is oriented towards performance improvement.
• Provides higher levels of service to Customers.
Benefits to the individual which in turn ultimately benefit the organisation:
• Helps the individual in making better decisions and effective problem solving.
• Aids in encouraging and achieving self-development and self-confidence.
• Helps a person handle stress, tension, frustration and conflict.
• Provides information for improving leadership knowledge, communication skills and attitudes.
• Increases job satisfaction and recognition.
• Moves trainee toward personal goals while improving interaction skills.
• Develops a sense of growth in learning.
• Helps a person develop speaking and listening skills; also writing skills when exercises are required.
• Helps eliminate fear in attempting new tasks.
Benefits in personnel relations, and policy implementation:
• Improves communication between groups and individuals.
• Aids in orientation for new employees and those taking new jobs through transfer or promotion.
• Improves interpersonal skills, which is a key for career and personal success.
• Makes organization policies, rules and regulations viable.
• Improves morale.
• Builds teamwork and cohesiveness in groups.
• Makes the organization a better place to work and live.
…And, as a result, increased profits.
Critical Examination of the Kirkpatrick Model
Despite being used to evaluate the training interventions in a wide variety of companies, different critiques have been made of the Kirkpatrick Model.
Most organizations carry out evaluation at the reactions level, some measure learning in technical skills, but few attempt to assess changes in behaviour or criteria of organizational effectiveness (Ralphs and Stephan, 1986; Saari et al., 1988; Training in Britain, 1989).
Alliger et al. (1997) carried out a meta-analysis of studies where reaction measures had been related to measures of learning (11 studies) and changes in behaviour (9 studies). They found that positive reactions did not predict learning gains better than negative ones (the average correlation between reactions and amount of learning was 0.02) nor were they any better at predicting changes in behaviour after the programme (correlation 0.07).
“The model does not appear to have a pre-training evaluation mechanism comprising an objective needs assessment framework, clear identification of learner needs, expectations and experience, and training design – all of which impact the four levels of the Kirkpatrick Model.” - Sumeet Varghese, HR Consultant
Bersin (2008) offers the following perspective:
General Principles of Training Measurement
• Measurement should deliver actionable information
• A Measurement Program should not be designed to cost-justify training
• Measure training as a support function
• A measurement program must meet the needs of multiple audiences
• Measurement should be a process, not a project
• The LMS is a foundation for measurement
• Dedicate resources
• Start simply and evolve over time.
The Pros and Cons of Using ROI
• ROI analysis assumes that training is treated as an investment
• In-depth ROI measurements are often difficult to believe
• In concept, the ROI of training should be extraordinarily high
• It is very difficult to correlate outcomes specifically to training
• How do you make ROI actionable?
• Use of “Potential ROI” during performance consulting
• “Performance-driven” vs. “Talent-driven” training
• Don’t let ROI become “Return on Insecurity”
Limitations of the Kirkpatrick Model
• The Kirkpatrick Levels are not a complete model
• The Model incorrectly positions the levels as a hierarchy
• The Kirkpatrick Model ignores the role of training as a service-delivery function
• Kirkpatrick misses other operational measures
The T.V. Rao Learning Systems Framework
The TVRLS Framework specifies 5 preconditions for transforming training from a cost into an investment: (1) The training program needs to be grounded in scientifically assessed training needs; (2) The participants want the training, or at least have a desire to learn (and aren’t just “training attendance” statistics!); (3) The training interventions are related to needs / competency gaps of the participants; (4) Participants draft action plans that transfer key learnings from the workshop to their workplace, and (5) the organization supports the implementation of the participants’ action plans through the provision of necessary resources (including monitoring and supervisory support) for their implementation.
If any of the five conditions specified in the TVRLS Framework are not met, the training becomes an expenditure rather than an investment!
The Rothwell et al Training / WLP Evaluation Model
In their book “What CEOs expect from Corporate Training” Rothwell, Lindholm and Wallick (Amacom, 2003) specify six dimensions for evaluating training / Workplace Learning and Performance (WLP):
1. How well did the intervention narrow the gap between “what is” and “What should be”?
2. How well did the interventions clarify the performance improvement objectives?
3. How well did the intervention serve to meet identified needs and achieve desired objectives?
4. What improvements if any resulted from the interventions?
5. What subsequent needs did the implementation process uncover?
6. How well did the workforce accept the intervention?
They also present nine questions to be asked in evaluating the support offered by the organization for WLP or training:
1. How well and how much does the organization support training and continuous learning?
2. How much and how well does your organization share information?
3. How much participation do employees have in decision that affects them?
4. How well does your organizational structure support continuous learning?
5. How well do employees and managers work together to achieve organizational goals through partnerships?
6. How much do the incentives provided promote effort to achieve organization results?
7. How secure managers and employees feel when faced with changes that affect their work?
8. How do employees and managers come to understand such change, reasons for change, and the opportunities in the change process for improved performance?
9. How much prominence is placed on innovation, risk and change? How supportive are the employees to change process?
Bringing Emotional Intelligence to the Workplace
Current interest in "emotional intelligence" has raised the question of whether it is possible to improve the social and emotional competence of adult workers. Research in training and development, sports psychology, and behavior change suggests that it is possible, but the typical approach used in corporate training programs usually is flawed. Social and emotional learning is different from cognitive and technical learning, and it requires a different approach to training and development.
The Success Case Method
Each year, organizations spend millions of dollars trying out new innovations and improvements—and millions will be wasted if they can’t quickly find out what’s working and what is not. The Success Case Method offers a breakthrough evaluation technique that is easier, faster, and cheaper than competing approaches, and produces compelling evidence decision-makers can actually use. The method has been proven effective in organizations ranging from Ford Motor Company to Gap International, Compaq, American Express, and the World Bank.
The Success Case Method is a carefully crafted, simple, and proven way of quickly finding out how well a new organizational initiative is working. Using the Success Case Method, people can get useful and accurate information about new initiatives: What results are being achieved using new approaches, what is working, what is not, and how it can be improved. The methods and tools in the Success Case Method can help change leaders, executives, managers, consultants, training directors, and anyone else who is trying to get things to work better.
Specifically, the Success Case Method gets information to help address four key questions about a new change initiative:
1. What is really happening? Who’s using what, and how well? Who’s not using things as planned? What’s getting used, and what isn’t? Which people and how many are having success? Which people and how many are not?
2. What results are being achieved? What good, if any, is being realized? What goals are being met? What goals are not? Is the change delivering the promised and hoped for results? What unintended results are happening?
3. What is the value of the results? What sort of dollar or other value can be placed on the results? Does the program appear to be worthwhile? Is it producing results worth more than its costs? What is its return on investment? How much more value could it produce if it were working better?
4. How can it be improved? What’s helping? What’s getting in the way? What could be done to get more people to use it? How can everyone be more like those few who are most successful?
Beyond Kirkpatrick: Adding New Dimensions to Training Evaluation
The American Heart Association (AHA) Training Evaluation Model offers additional dimensions that go beyond the Kirkpatrick Model:
Level 1 = Course: Evaluates the training based on content, structure, methods, materials and delivery.
Level 2 = Satisfaction (Happiness Sheets): Assesses trainees feelings about the trainer, quality of material presented, methods of presentation and environment (eg. room temperature).
Level 3 = Opinion: Assesses trainees’ attitudes / perceptions about utilization of training, own learning, trainings’ value.
Level 4 = Knowledge Acquision: Assesses learning and recalling terms, definitions and facts.
Level 5 = Knowledge comprehension: Assesses understanding concepts and relationships, recognizing examples in practice, and problem solving.
Level 6 = Skill Demonstration: Assesses application of learned material to new and concrete situations within the training environment.
Level 7 = Skill Transfer: Assesses application of new knowledge and skills outside the classroom and on the job.
Level 8 = Agency / Organisational Impact
Level 9 = Client Outcomes
Level 10 = Community Impacts.
Conclusions
Although the Kirkpatrick Model is widely used and influential, its shortcomings are likely to accelerate the search and adoption of alternative models for assessing the return on investment for training.
Shortcomings of the Kirkpatrick Model include the omission of a pre-training evaluation mechanism, as well as the recognition that the training can have a broader beneficial impact on the participant’s family and community, which is currently not recognized or measured in the Kirkpatrick Model.
Promising contenders to augment or replace the Kirkpatrick Model include the T.V. Rao Learning Systems Framework, The Rothwell et al Training / Work Place Learning Evaluation Model, the Success Case Method, Goleman’s “Optimal Process” model, and the American Heart Association Training Evaluation Model that specifies Client Outcomes and Community Impacts, in addition to Organisational Impact.
Bibliography
Daniel Goleman, Cary Cherniss, Robert Emmerling, Kimberly Cowan and Mitchel Adler (1998), Bringing Emotional Intelligence to the Workplace, The Consortium for Research on Emotional Intelligence in Organisations
Imon Ghosh (2004), The Benefits of Training, Newstalk, vol 12 no. 3
Jack J Phillips (2005), Investing In Your Company’s Human Capital, Amacom Books.
Jane Bozarth (2008), From Analysis To Evaluation: Tools, Tips and Techniques for Trainers, Pfeiffer
Josh Bersin (2008), The Training Measurement Book: Best Practices, Proven Methodologies, and Practical Tips, Pfeiffer
Kaliym A. Islam (2006), Developing and Measuring Training the Six Sigma Way, Pfeiffer
Paul Fairhurst and Debbie Carter (2008), Exploring The Future of Workplace Learning 2020, Training Journal and The Institute for Employment Studies.
Peter Bramley (1999), Evaluating effective management learning, Journal of European Industrial Training, pp 145 – 153
Robert O. Brinkerhoff (2003), The Success Case Method: Find Out Quickly What's Working and What's Not, Berrett-Koehler Publishers
T.V. Rao (1998), Converting Training Expenses into Investments: TVRLS Formula; Towards 2020, No.2, February 1998, Pages 2-3.
William J. Rothwell, John E. Lindholm, William G. Wallick (2003), What CEOs Expect from Corporate Training, Amacom Books
Imon Ghosh is Training Manager at METRO Cash & Carry India
