Measuring Training Impact

Most L&D professionals learned the Kirkpatrick Model early on. Fewer have seen it applied beyond Level 1. Here's what each level can actually look like when you put it into practice, not just the textbook definition. ✨ Level 1: Reaction 🔹 Textbook version: Did learners find the training engaging and worth their time? ✅ In practice: Instead of "Did you enjoy this session?", ask "Was this relevant to the work you do?" and "Could you apply this right away?" ✅ Metric to track: Relevance and applicability ratings, not just satisfaction scores. ✨ Level 2: Learning 🔹 Textbook version: Did learners gain the intended knowledge or skills? ✅ In practice: Replace recall-based quizzes with scenario-based checks. Can the learner apply the concept to a situation they'd actually face? ✅ Metric to track: Pre/post assessment scores on scenario-based questions, not just "did you pass the quiz." ✨ Level 3: Behavior 🔹 Textbook version: Are learners applying what they learned on the job? ✅ In practice: 30/60/90-day check-ins, manager observations, or peer feedback on whether the new behavior is showing up in real work. ✅ Metric to track: % of participants demonstrating the target behavior, based on manager or peer input, not self-reported confidence. ✨ Level 4: Results 🔹 Textbook version: Did the training impact business outcomes? ✅ In practice: Pick one business metric the program was meant to influence, before you build it, not after, and track the change. ✅ Metric to track: Movement in that specific KPI (error rates, time-to-productivity, conversion rates, retention) compared to a baseline. Most programs are measured thoroughly at Level 1 and barely at all beyond it. But Levels 3 and 4 are where the "did this actually matter" conversation happens, and they are also where L&D earns a seat at the table. Which level does your organisation measure consistently, and which one do you wish you could measure better? #LearningAndDevelopment #LnD #KirkpatrickModel #TrainingEvaluation #InstructionalDesign #LearningMeasurement #TrainingAndDevelopment #LnDStrategy

Training LLMs for spam classification: I added 14 experiments comparing different approaches: https://lnkd.in/gTNVvGcj - which token to train - which layers to train - different model sizes - LoRA - unmasking - and more! Any additional experiments you'd like to see? And here are the take aways for the table shown in the picture: 1. Training the Last vs. First Output Token (Row 1 vs. 2): Training the last output token results in substantially better performance compared to the first. This improvement is expected due to the causal self-attention mask. 2. Training the Last Transformer Block vs. Last Layer (Row 1 vs. 3): Training the entire last transformer block is also results in substantially better results than training only the last layer. 3. Training All Layers vs. Last Transformer Block (Row 1 vs. 4): Training all layers shows a modest improvement of ~2% over just training the last transformer block, but it requires almost three times longer in terms of training duration. 4. Using Larger Pretrained Models (Row 1 vs 5, and Row 1 vs. 6 and 7): Employing a 3x larger pretrained model leads to worse results. However, using a 5x larger model improves performance compared to the initial model, as was anticipated. Similarly, the 12x larger model improves the predictive performance even further. (The medium model was perhaps not well pretrained or the particular finetuning configuration works not as well for this model.) 5. Using a Model with Random Weights vs. Pretrained Weights (Row 1 vs. 8): Utilizing a model with random weights yields results that are only slightly worse by 1.3% compared to using pretrained weights. 6. Using LoRA (Low-Rank Adaptation) vs Training All Layers (Row 9 vs. 4): Keeping the model frozen and adding trainable LoRA layers (see Appendix E for details) is a viable alternative to training all model parameters and even improves the performance by 1% point. As it can be seen by the 1% lower gap between the training and validation accuracy when using LoRA, this is likely due to less overfitting. 7. Padding Input to Full Context Length vs. Longest Training Example (Row 1 vs. 10): Padding the input to the full supported context length results is significantly worse. 8. Padding vs no padding (Row 1 vs. 11 and 12): The `--no_padding` option disables the padding in the dataset, which requires training the model with a batch size of 1 since the inputs have variable lengths. This results in a better test accuracy but takes longer to train. In row 12, we additionally enable gradient accumulation with 8 steps to achieve the same batch size as in the other experiments. 9. Disabling the causal attention mask (Row 1 vs. 13): Disables the causal attention mask used in the multi-head attention module. This means all tokens can attend all other tokens. The model accuracy is slightly improved compared to the GPT model with causal mask.

Most training programs create excitement. Very few create measurable business impact. A few months ago, I worked with an organization that had a very specific challenge. Their frontline teams were attending workshops, feeling motivated, taking notes but when it came to actual performance on the field, their sales conversion was very low. Great energy. Poor execution. Something was missing. So before designing the learning intervention, I asked one simple question: “What’s the real context in which your people operate daily?” Not the role. Not the job description. Not the competencies. The context. What pressures do they face? What conversations are toughest? Where do deals collapse? Who influences decisions? What behaviours matter most on the ground? The organization opened up. We mapped real scenarios. We shadowed calls. We watched interactions. We decoded customer psychology. We understood the reality behind the numbers. Only then did we build the training journey. Not generic content. Not textbook concepts. Not motivational theory. But a program designed exactly around their on-ground realities. The impact. Over the next eight weeks, something changed. Sales conversations became sharper. Objections were handled with more confidence. Teams spoke value, not price. Managers reinforced learning consistently. The conversion saw a huge jump and this was created not by more training, but by the right training. The lesson is simple: Content informs. Context transforms. Workshops don’t create results. Relevance does. When learning mirrors the real world, people don’t just listen they apply. When they apply, organizations grow. What’s one area in your team where you feel content is high but context is missing? If your organization wants training that delivers real, measurable outcomes let’s talk.

❗ Only 12% of employees apply new skills learned in L&D programs to their jobs (HBR).  ❗ Are you confident that your Learning and Development initiatives are part of that 12%? And do you have the data to back it up?  ❗ L&D professionals who can track the business results of their programs report having a higher satisfaction with their services, more executive support and continued and increased resources for L&D investments.    Learning is always specific to each employee and requires personal context. Evaluating training effectiveness shows you how useful your current training offerings are and how you can improve them in the future. What’s more, effective training leads to higher employee performance and satisfaction, boosts team morale, and increases your return on investment (ROI). As a business, you’re investing valuable resources in your training programs, so it’s imperative that you regularly identify what’s working, what’s not, why, and how to keep improving. To identify the Right Employee Training Metrics for Your Training Program, here are a few important pointers: ✅ Consult with key stakeholders – before development, on the metrics they care about. Make sure to use your L&D expertise to inform your collaboration. ✅Avoid using L&D jargon when collaborating with stakeholders – Modify your language to suit the audience. ✅Determine the value of measuring the effectiveness of a training program. It takes effort to evaluate training effectiveness, and those that support key strategic outcomes should be the focus of your training metrics. ✅Avoid highlighting low-level metrics, such as enrollment and completion rates. 9 Examples of Commonly Used Training Metrics and L&D Metrics 📌 Completion Rates: The percentage of employees who successfully complete the training program. 📌Knowledge Retention: Measured through pre- and post-training assessments to evaluate how much information participants have retained. 📌Skill Improvement: Assessed through practical tests or simulations to determine how effectively the training has improved specific skills. 📌Behavioral Changes: Observing changes in employee behavior in the workplace that can be attributed to the training. 📌Employee Engagement: Employee feedback and surveys post-training to assess their engagement and satisfaction with the training. 📌Return on Investment (ROI): Calculating the financial return on investment from the training, considering costs vs. benefits. 📌Application of Skills: Evaluating how effectively employees are applying new skills or knowledge in their day-to-day work. 📌Training Cost per Employee: Calculating the total cost of training per participant. 📌Employee Turnover Rates: Assessing whether the training has an impact on employee retention and turnover rates. Let's discuss in comments which training metrics are you using and your experience of using it. #MeetaMeraki #Trainingeffectiveness

Most Train-the-Trainer programmes fail for one simple reason... Transfer is assumed, not designed. A new paper in the International Journal of Training and Development finally tackles a long-standing blind spot in L&D: 👉 How trainers themselves actually learn , and why that learning so often fails to show up in practice. Wisshak et al. (2025) propose a generic “offer-and-use” model for Train-the-Trainer programmes, adapted from teacher education and grounded in decades of transfer research. Training effectiveness is not determined by what is offered, but by how trainers perceive, interpret, and use learning opportunities within their real work context. The model highlights six interacting elements: • Training design & facilitation quality • Individual trainer factors (motivation, self-efficacy, prior knowledge) • Contextual factors (support, culture, opportunity to apply) • Perceived relevance and engagement • Actual learning processes • Outcomes, with transfer (behaviour change) as the non-negotiable criterion What I find particularly important is this: Many trainers are self-employed or freelance, yet most transfer models assume a supportive organisation, manager reinforcement, and stable teams. This paper explicitly addresses that mismatch, suggesting peer networks, follow-ups, feedback loops, and deliberate transfer scaffolding. Implication for L&D: If your Train-the-Trainer programme is evaluated mainly on satisfaction scores or content coverage, you are measuring the least predictive indicators of success. Transfer isn’t a phase. It’s a system property.

Are we drinking from a dirty stream of information? That’s what it feels like when the Internet feeds AI and AI pours it right back into the Internet. Bad data becomes content. That content becomes training data. And the cycle keeps circulating, only murkier each time. We don’t need more data. We need cleaner data. 🧩 Because the quality of what AI produces is only as pure as the source it drinks from. If you’re in charge of workforce development, whether as a CEO, business owner, HR leader, or like me, in Learning and Development, you have to be intentional about what gets put out there. There’s a reason we rely on subject matter experts (SMEs). But lately, I’ve seen more people skip that step trusting AI to fill in the blanks. Here’s the problem: AI doesn’t understand, it predicts. And when it predicts wrong, your people learn wrong. Before launching any AI-generated learning material, communication, or process guide: ✅ Validate it with an SME using the 3R Method: 1. Relevance – Does this align with how we actually do things in our organization? AI might sound polished, but it can invent steps that don’t exist or skip critical compliance details. 2. Risk – Could this information create confusion, errors, or compliance issues if followed as written? An SME can quickly spot risks that AI misses, especially in regulated or safety-sensitive environments. 3. Reflection – Does this reflect our tone, culture, and way of communicating? AI often defaults to generic phrasing. SMEs help make it sound like your company’s voice, not a machine’s. Your workforce learns from the content you approve. Make sure it’s something you’d be proud to stand behind. Your AI is only as good as the expertise that validates it. 🌱 More on AI + Workforce Development → Janet Perez

Constant workflow evaluation is crucial to meet business demands. A recent leadership training reshaped my approach, stressing the importance of questioning norms and assessing if traditions still add value. One story that perfectly captures the essence of this training is the parable of the soldier’s barracks and the parade slab. Imagine a military base decades ago where soldiers laid a concrete slab to hold parades. However, before the cement dried, animals would often trample on it, creating an unsightly mess. So, a soldier was assigned to guard the slab at night, preventing any intrusion until it dried completely. But over the years, this nighttime guarding became a routine task, regardless of necessity or even the slab’s condition. The soldiers rotated nightly shifts to guard this parade slab—an unexamined duty passed down through generations. One day, a recruit questioned the reason behind guarding this slab. Strangely, nobody knew why they were guarding it, nor could they remember when the slab was last poured. The original purpose had long since faded, leaving only an empty ritual that served no purpose, other than occupying valuable time and resources. This example resonated with me deeply. How often do we continue tasks and workflows because “that’s just how it’s always been done”? Just like the soldiers in the barracks, we may be blindly guarding proverbial slabs that have long outlived their relevance. In our quest to become more productive and cost-effective, these "slabs" need to be identified and eliminated. The training encouraged steps to dismantle workflows and streamline processes: Map Out the Process Chart each action and person involved to expose redundancies and tasks done out of habit, not purpose. Define Purpose for Each Step Ask, “What’s the intended outcome?” Many tasks are formalities with no impact. Engage Team Members Team feedback reveals inefficiencies leaders may overlook. Front-line employees often see issues we don’t as most leaders. Use a “What if” Mindset Boldly ask, “What if we didn’t do this at all?” Challenge task necessity. Implement and Track Testing changes and measuring outcomes ensures productivity gains are tangible. The results: reduced non-value tasks and measurable cost savings. Outdated workflows can waste up to 20% of productive time. Morale also suffers when employees perform pointless tasks. A lasting lesson was that productivity comes from fostering a culture of inquiry. Leaders aren’t just problem solvers; they’re problem finders, willing to challenge even the most accepted routines. Tradition can be comforting, but in business, clinging to unnecessary tasks is an expense we can’t afford. This experience taught me to always ask, “Why are we doing this?” If the answer doesn’t align with our goals, it’s time to break the mold and let go of practices that don’t serve us and the business. By embracing inquiry and challenging norms, we build agile and resilient organizations

How do we know if RL is going well or not? Here are some key health indicators to monitor during the RL training process… RL is a complex process made up of multiple disjoint systems. It is also computationally expensive, which means that tuning / debugging is expensive too! To quickly identify issues and iterate on our RL training setup, we need intermediate metrics to efficiently monitor the health of the training process. Key training / policy metrics to monitor include: (1) Response length should increase during reasoning RL as the policy learns how to effectively leverage its long CoT. Average response length is closely related to training stability, but response length does not always monotonically increase—it may stagnate or even decrease. Excessively long response lengths are also a symptom of a faulty RL setup. (2) Training reward should increase in a stable manner throughout training. A noisy or chaotic reward curve is a clear sign of an issue in our RL setup. However, training rewards do not always accurately reflect the model’s performance on held-out data—RL tends to overfit to the training set. (3) Entropy of the policy’s next token prediction distribution serves as a proxy for exploration during RL training. We want entropy to lie in a reasonable range—not too low and not too high. Low entropy means that the next token distribution is too sharp (i.e., all probability is assigned to a single token), which limits exploration. On the other hand, entropy that is too high may indicate the policy is just outputting gibberish. Similarly to entropy, we can also monitor the model’s generation probabilities during RL training. (4) Held-out evaluation should be performed to track our policy’s performance (e.g., average reward or accuracy) as training progresses. Performance should be monitored specifically on held-out validation data to ensure that no reward hacking is taking place. This validation set can be kept (relatively) small to avoid reducing the efficiency of the training process. An example plot of these key intermediate metrics throughout the RL training process from DAPO is provided in the attached image. To iterate upon our RL training setup, we should i) begin with a reasonable setup known to work well, ii) apply interventions to this setup, and iii) monitor these metrics for positive or negative impact.

Employees don’t hate training. They hate training that wastes their time. I’ve seen highly motivated, curious people disengage not because they didn’t care, but because the learning felt disconnected from reality. When learning is something done to people rather than done with them, resistance is a rational response. The format rarely matters. I’ve seen brilliant results from digital, face-to-face, blended and social approaches and equally poor results from all of them too. The difference was always whether the experience created meaning, relevance and momentum in real work. Most L&D problems aren’t learning problems. They’re work problems that learning is being asked to fix after the fact. We design programmes around what people should know, not what they need to do differently on Monday morning. Then we’re surprised when nothing changes. Relevance isn’t a nice-to-have. It’s the entry ticket. If people can’t immediately see how learning helps them hit targets, handle pressure, save time or avoid mistakes, you’ve already lost them. Performance improves when learning... ↳ Starts with real work, not content ↳ Solves a problem people actually have ↳ Is applied immediately, not “later” ↳ Is supported by managers, not just L&D ↳ Is measured by behaviour change, not completion rates This is where L&D often gets stuck. We optimise for delivery instead of impact. We protect programmes instead of questioning them. We report activity instead of outcomes. If learning doesn’t change decisions, actions or results, it’s just organised distraction. People don’t disengage from learning. They disengage from irrelevance. And until L&D shifts from “Did they attend?” to “Did anything change?”, nothing else really matters! What's your take on this? ---------------------------------- Follow me at Sean McPheat for more L&D content and and then hit the 🔔 button to stay updated on my future posts. ♻️ Save for later and repost to help others.

Companies spend millions on sales training. But less than 1 in 10 dollars goes to knowing if it worked. In addition, nearly 1 in 3 companies run zero formal evaluation at all. That's what the research says – and it reflects what many of us have felt in the room: ✅ We ran the training. ❓But did it actually work? As enablement professionals, we’re often caught between anecdotes and dashboards. Between sales spikes that may or may not be linked to our efforts and gut instincts that can’t hold up in a boardroom. We need to move from guesswork to genuine insight. That’s why I wrote a deep-dive on sales training evaluation: what the research says, and which models actually work in practice. --- In my new guide, I break down the five most effective models for evaluating training impact: 🔹 Kirkpatrick Model – the classic 4-level framework 🔹 Phillips ROI Model – adds ROI calculation to Kirkpatrick 🔹 New World Kirkpatrick – repositions ROI as Return on Expectations 🔹 Brinkerhoff’s Success Case Method – focuses on extremes to find truth 🔹 LTEM (Learning Transfer Evaluation Model) – the most diagnostic model out there And, I cover five honourable mentions worth exploring: 🔸 CIPP Model – evaluates context, inputs, process, and product 🔸 COM-B Model – breaks down behaviour change 🔸 6Ds – emphasises reinforcement beyond the classroom 🔸 Bersin’s Impact Measurement Framework – business-linked metrics 🔸 Anderson Model – ties training to strategic priorities Whether you're launching a new programme or defending your budget, this will give you a sharper lens and a stronger voice. --- 📌 Want access to the high-res one-pager + full guide? Comment “sales training evaluation” and I’ll DM it to you. Let’s raise the bar for what enablement can prove and improve. ✌️ #sales #salesenablement #salestraining