Quick Answer
What is the Kirkpatrick Model?
The Kirkpatrick Model is the most widely used framework for training evaluation. Developed by Donald Kirkpatrick in 1959, it measures training effectiveness across four levels: Level 1 (Reaction) – how learners respond to the training; Level 2 (Learning) – what knowledge or skills they gained; Level 3 (Behavior) – whether they apply the learning on the job; and Level 4 (Results) – what measurable business impact the training produced. The Kirkpatrick model of training evaluation is used by L&D teams to move beyond completion rates and demonstrate whether training changed performance and delivered business value. Evaluation planning should begin before the training is built – not after it has been delivered.
- Level 1 – Reaction: Learner satisfaction and perceived relevance
- Level 2 – Learning: Knowledge, skills, or attitude acquired
- Level 3 – Behavior: On-the-job application of learning
- Level 4 – Results: Business impact linked to training
What Is the Kirkpatrick Model of Training Evaluation?
The Kirkpatrick Model is a four-level framework used to evaluate the effectiveness of training programs. It provides a structured way for L&D teams to measure not just whether training happened but whether it produced the outcomes it was designed to produce.
The core problem the Kirkpatrick evaluation model solves is the gap between activity metrics and outcome metrics. Most organizations can easily report how many employees completed a training course, how long they spent on it, and what score they achieved. What they struggle to report is whether the training changed how those employees perform on the job and whether that change produced a measurable business result. The Kirkpatrick model provides the framework for answering those harder questions systematically.
Each level of the Kirkpatrick learning model builds on the previous one. Positive learner reactions do not guarantee that learning occurred. Demonstrated learning gains do not guarantee that behavior changed. Behavior change does not by itself confirm business impact. The four levels together provide a complete picture of training effectiveness that no single data point can provide on its own.
The model is also known as the Kirkpatrick model of evaluation, the four levels of the Kirkpatrick model, and the Kirkpatrick training evaluation framework. All of these terms refer to the same framework introduced by Donald Kirkpatrick and expanded by his son James Kirkpatrick in subsequent decades.
Origins – Who Developed the Kirkpatrick Model?
Donald Kirkpatrick introduced the Kirkpatrick Model in 1959 through a series of articles in the Journal of the American Society of Training Directors. At the time, training evaluation in corporate settings was largely limited to post-training satisfaction surveys – what practitioners called “smile sheets.” Kirkpatrick argued that effective evaluation required looking beyond learner satisfaction to ask whether training produced genuine learning, changed on-the-job behavior, and ultimately improved organizational performance.
Kirkpatrick formalized and expanded the framework in his 1994 book Evaluating Training Programs: The Four Levels, which remains the definitive text on the model. His son James Kirkpatrick subsequently expanded the framework with the New World Kirkpatrick Model a refinement that emphasized beginning evaluation planning at Level 4 (Results) and working backward, ensuring that every training design decision connects to the business outcome the training is meant to support.
The model’s longevity over 65 years of continuous use in corporate training comes from its simplicity and practical utility. Despite the emergence of more sophisticated evaluation methodologies, the Kirkpatrick model of training evaluation remains the most widely referenced framework in corporate L&D globally, cited in ATD research, CIPD guidance, and organizational learning strategies across industries.
The Four Levels of the Kirkpatrick Model – In Depth
| Level | What It Measures | Key Questions | Data Sources |
|---|---|---|---|
| Level 1 — Reaction | Learner satisfaction and perceived relevance | Was the training relevant to my role? Did the format support learning? Was the content clear? | Post-training surveys, feedback forms, LMS satisfaction ratings |
| Level 2 — Learning | Knowledge, skills, or attitude acquired | Did learners gain the intended knowledge? Can they demonstrate the skill? Did their confidence change? | Assessments, quizzes, simulations, pre/post tests, skill demonstrations |
| Level 3 — Behavior | On-the-job application of learning | Are learners applying what they learned? Has behavior changed 30/60/90 days post-training? | Manager observations, performance reviews, 360 feedback, follow-up assessments |
| Level 4 — Results | Business impact linked to training | Did training contribute to measurable improvement in the KPI it was designed to address? | KPI data, productivity metrics, error rates, compliance incidents, revenue, customer satisfaction |
Level 1 – Reaction
Level 1 of the Kirkpatrick model measures how learners respond to the training experience their satisfaction, perceived relevance, and engagement. This is the most commonly collected evaluation data in corporate training, primarily because it is the easiest to gather: a post-training survey immediately after course completion captures the data while the learner’s experience is fresh.
Kirkpatrick distinguished between two types of reaction measurement: satisfaction (did learners enjoy or appreciate the training?) and relevance (did learners see the connection between the training content and their actual job performance?). Both matter, but relevance is the more predictive signal. High satisfaction with low perceived relevance is a warning sign learners found the training enjoyable but disconnected from their work, which predicts poor Level 3 behavior transfer. High relevance scores, even with moderate satisfaction, predict stronger application.
Useful Level 1 survey questions go beyond generic satisfaction ratings. Effective reaction measurement asks: “Was the content directly applicable to situations you face in your role?” “What would you change about the training to make it more useful?” “Do you feel prepared to apply what you learned?” These questions surface actionable improvement data that generic 5-star satisfaction ratings do not. An LMS with built-in survey tools deploys Level 1 data collection automatically after course completion and stores responses for trend analysis across the learner population.
Critical caution: Level 1 data should never be used as the primary evidence of training effectiveness. Learners can give high satisfaction ratings to a course that changes none of their on-the-job behaviors. Positive reaction scores are a necessary but not sufficient condition for effective training. They confirm the learning experience was well-designed and relevant they do not confirm that learning occurred.
Level 2 – Learning
Level 2 of the Kirkpatrick model measures what learners actually gained from the training the knowledge they acquired, the skills they developed, or the attitudes that shifted. This is the level where assessment data from the training itself becomes the primary evidence.
The Kirkpatrick framework distinguishes three categories of learning outcomes: knowledge (what learners now know that they did not know before), skills (what they can now do that they could not do before), and attitude (how their perspective or confidence regarding a topic has changed). Different training objectives require different Level 2 measurement approaches:
- Knowledge gains are measured through assessments pre/post tests, quizzes, scenario-based questions that ask learners to apply knowledge to realistic situations rather than simply recall definitions.
- Skill development is measured through demonstrations, simulations, or practical tasks that require learners to perform the skill under observation not just describe it. An AI roleplay simulation where a sales rep practices objection handling measures skill acquisition far more reliably than a multiple-choice quiz asking what they should say.
- Attitude change is measured through pre/post surveys asking learners to rate their confidence, perceived competence, or intention to apply specific behaviors.
The assessment design decision is the most important Level 2 choice an instructional designer makes. Assessments that test recall of slide content measure whether learners remember the training, not whether they can apply it. Questions that present realistic work scenarios and ask learners to choose the correct action measure whether learning has been internalized at the application level which is far more predictive of Level 3 behavior change. An eLearning authoring tool that supports scenario-based question formats makes this distinction practical to implement at scale.
The LMS tracks all Level 2 data automatically assessment scores, pass/fail rates, number of attempts, time spent on assessments, and score distributions across the learner population. This data identifies which concepts learners struggle with most, which allows instructional designers to update content before the next cohort completes the course rather than after.
Level 3 – Behavior
Level 3 of the Kirkpatrick model is where most training evaluation programs break down and where the most important development question gets answered. Level 3 asks: are learners actually doing the thing the training was designed to get them to do, in their real work context, after the training is over?
This is fundamentally different from knowing whether they passed an assessment. A learner can score 95% on a compliance assessment and return to work and continue performing the non-compliant behavior they were doing before the training because the assessment tested their ability to recognize the correct answer in a multiple-choice format, not their commitment to performing differently in a real situation where the behavior is habitual and the consequences are not immediately visible.
Level 3 measurement requires data sources outside the LMS. Manager observation is the most direct: structured observation checklists used by managers in the 30, 60, and 90 days following training provide direct evidence of behavioral change. Performance data error rates, compliance audit results, quality scores, call handling metrics provides objective evidence where the behavior is observable in operational systems. Follow-up assessments 60–90 days post-training measure whether knowledge has been retained; combined with performance data, they provide a more complete picture than either alone.
The New World Kirkpatrick Model adds an important concept to Level 3: required drivers. Even when learners have the knowledge and intention to apply their learning, behavior transfer often fails because of environmental factors managers who do not reinforce the new behavior, processes that make the old behavior easier, incentive structures that reward the behavior the training was trying to change. Required drivers are the organizational conditions (manager reinforcement, peer support, process alignment, performance accountability) that make behavior transfer possible. Designing these into the program not assuming they exist is the difference between training that transfers and training that fades.
Level 4 – Results
Level 4 of the Kirkpatrick model measures the business impact that the training was ultimately designed to produce the organizational outcomes that justified the training investment in the first place. This is the level that matters most to senior business leaders, and the level that most L&D teams measure least rigorously.
Level 4 results are the KPIs or business metrics the training was designed to move. Examples by training type: compliance training → reduction in regulatory incidents, audit findings, or near-miss reports; sales training → conversion rate improvement, average deal size, time to first close; customer service training → customer satisfaction scores, escalation rates, first-contact resolution rates; onboarding training → time-to-productivity, 90-day retention rate; safety training → incident rate, lost-time injury frequency.
The Level 4 measurement challenge is attribution business outcomes are influenced by many factors simultaneously, and isolating the training’s specific contribution is genuinely difficult. A company whose sales revenue increases after a sales training program cannot definitively prove the training caused the improvement without a control group methodology that most corporate environments cannot practically implement. The New World Kirkpatrick Model is candid about this: the goal of Level 4 is not to prove causation but to gather credible evidence of correlation between training activity and business outcome movement, combined with qualitative confirmation from managers that the behavior changes observed at Level 3 are connected to the result changes observed at Level 4.
The most important Level 4 principle: The business outcome the training is meant to influence should be identified and baselined before the training is built not discovered after delivery. An L&D team that builds a compliance training program and then attempts to identify what business metric it should have been designed to improve is starting the evaluation in the wrong order. Starting with the Level 4 result and designing backward through Levels 3, 2, and 1 is the approach James Kirkpatrick’s New World model formalizes and what the most effective corporate L&D evaluations consistently apply.
Kirkpatrick Model Examples by Training Type
The following Kirkpatrick model examples show how each level applies in practice across common corporate training contexts:
| Training Type | Level 1 — Reaction | Level 2 — Learning | Level 3 — Behavior | Level 4 — Results |
|---|---|---|---|---|
| Compliance Training | Post-module relevance survey | Scenario-based assessment pass rate | Manager observation: are procedures being followed? Compliance audit findings | Regulatory incident rate, audit pass rate, near-miss reports |
| Sales Training | Perceived applicability rating | Roleplay or simulation score; objection handling assessment | Manager call reviews: are reps using the new methodology? Pipeline activity | Conversion rate, average deal size, win rate, quota attainment |
| Onboarding | New hire satisfaction survey (Day 30) | Role-readiness assessment at Day 30 and Day 60 | Manager assessment: performing independently? Making correct decisions? | Time-to-productivity, 90-day retention rate, error rates in first 3 months |
| Customer Service | Perceived usefulness of scenarios | Scenario quiz scores; call simulation performance | Call quality scores, escalation frequency, supervisor observations | CSAT scores, first-contact resolution rate, average handle time |
| Safety Training | Relevance to daily work activities | Safety procedure knowledge assessment; practical demonstration | Safety walk observations, near-miss reporting behavior, PPE compliance | Incident rate, lost-time injury frequency, workers’ compensation costs |
| Leadership Development | Perceived relevance to current challenges | Pre/post confidence self-assessment; scenario-based leadership decisions | 360-degree feedback at 90 days; team engagement survey data | Team retention rate, employee engagement scores, manager effectiveness ratings |
How to Apply the Kirkpatrick Model – Step by Step
Step 1 – Start with the Level 4 outcome
Before building any training program, identify the specific business result the training is designed to improve. Not “improve compliance” that is a category. Identify the specific KPI: “reduce the rate of data handling incidents from 12 per quarter to fewer than 4.” The specificity of the Level 4 target determines the precision of every design decision that follows. A training program designed to move a specific measurable KPI is fundamentally different from a training program designed to “cover the compliance requirements.”
Step 2 – Define Level 3 behaviors
Ask: what do employees need to do differently on the job to produce the Level 4 result? These are the Level 3 target behaviors the specific observable actions the training must change. For the data handling compliance example: employees must correctly classify data before storing or sharing it, must use the approved encryption method for external transfers, and must report any potential incident within 4 hours of discovery. Each of these is a specific, observable behavior that can be measured after training. These behaviors become the learning objectives.
Step 3 – Design Level 2 assessment against Level 3 behaviors
Write assessment questions that test whether learners can perform the Level 3 behaviors, not whether they can recall the training content. For the data handling example: present a realistic scenario (“you receive a request to email a customer file to an external consultant what do you do?”) and require learners to choose the correct action from realistic options. Assessment designed to predict Level 3 behavior transfer performs significantly better than assessment designed to confirm content recall.
Step 4 – Plan Level 1 measurement before building the course
Write the Level 1 survey questions before content development begins, not after. Focus survey questions on relevance, applicability, and preparation for specific work situations not on satisfaction with the content format or instructor quality. Embed the survey in the LMS so it deploys automatically after completion. Decide in advance what Level 1 threshold would trigger a course content review.
Step 5 – Define Level 3 measurement method and timeline
Before launch, agree with the business sponsor on how Level 3 behavior change will be measured and when. Will managers complete an observation checklist at 30 days? Will performance data be pulled from operational systems at 60 and 90 days? Who owns the data collection? Leaving Level 3 measurement undefined until after training is delivered almost always means it never happens.
Step 6 – Establish Level 4 baseline
Record the current KPI value before training launches. This is the baseline against which post-training performance will be compared. Without this baseline, demonstrating Level 4 impact becomes impossible regardless of how well the training performed.
Step 7 – Deliver training and collect Level 1 and 2 data
Deploy the training through the LMS. Collect Level 1 survey responses automatically. Review Level 2 assessment score distributions within the first week of deployment if a significant proportion of learners are failing specific questions, the content for that topic needs review before more learners complete the course.
Step 8 – Collect Level 3 data at defined intervals
At 30, 60, and 90 days post-training, collect the agreed Level 3 data. Compare manager observations and performance metrics against the Level 3 target behaviors defined in Step 2. Identify where behavior transfer is occurring and where it is not. Investigate environmental barriers (Required Drivers) when behavior has not changed despite strong Level 2 scores.
Step 9 – Measure Level 4 KPI movement
At 90 days and 6 months post-training, compare the target KPI against the baseline established before training. Combine this quantitative data with qualitative input from managers and business leaders about whether they attribute the observed performance change to the training program. Present both together to business leadership.
Kirkpatrick Model vs Other Training Evaluation Frameworks
The Kirkpatrick model of training evaluation is the most widely used but not the only evaluation framework. Understanding how it compares to other approaches helps L&D teams choose the right methodology for their context:
| Framework | Core Approach | Strengths | Limitations | Best For |
|---|---|---|---|---|
| Kirkpatrick Model | Four-level hierarchy: Reaction → Learning → Behavior → Results | Simple, practical, widely understood, applicable across all training types | No direct ROI calculation; attribution at Level 4 is difficult | Most corporate training programs — especially where behavioral outcomes matter most |
| Phillips ROI Model | Extends Kirkpatrick by adding a fifth level — financial ROI calculation (benefits vs costs) | Produces a monetary ROI figure; supports investment justification conversations | Resource-intensive; many training benefits resist monetization; can produce misleading precision | High-cost programs where financial justification is required by senior leadership |
| CIPP Model (Stufflebeam) | Context, Input, Process, Product — evaluates program design and implementation quality, not just outcomes | Supports formative evaluation (improving programs during delivery); comprehensive program quality view | Complex and resource-intensive; less outcome-focused than Kirkpatrick | Large-scale programs where continuous improvement during delivery is a priority |
| Brinkerhoff Success Case Method | Identifies the most and least successful training participants and studies what made the difference | Efficient; surfaces organizational barriers to transfer; generates compelling success stories | Not a comprehensive population measurement; selected cases may not represent average outcomes | Programs where understanding why transfer succeeds or fails is the primary question |
In practice, most corporate L&D teams use the Kirkpatrick model as their foundational framework and add Phillips ROI methodology selectively for high-cost or high-stakes programs where financial justification is required. The CIPP model is more common in public sector and educational contexts than in corporate training.
Kirkpatrick Model Limitations and Criticisms
An honest treatment of the Kirkpatrick levels of evaluation requires acknowledging where the framework has limitations:
The attribution problem at Level 4
Connecting training activity to business outcomes in a way that establishes causation rather than correlation is genuinely difficult in most organizational contexts. If customer satisfaction improves following customer service training, many factors may have contributed staff turnover, product improvements, pricing changes, management quality, economic conditions. Claiming the training caused the improvement without supporting evidence overstates what the data shows. The Kirkpatrick framework acknowledges this but does not resolve it — it provides a structure for gathering evidence, not a methodology for proving causation.
Level 3 measurement is rarely completed
Research consistently shows that while most organizations collect Level 1 and Level 2 data routinely, Level 3 measurement is conducted for fewer than 15% of training programs in most corporate L&D functions. The reasons are practical: Level 3 data requires collaboration with managers and business operations teams, takes 30–90 days to become available, and requires effort that falls outside the LMS’s automatic reporting capability. The result is that most training evaluation stops where the data is easiest to collect rather than where the most important evidence exists.
The hierarchy assumption
The Kirkpatrick model assumes that Levels 1–4 are hierarchically linked that positive reaction produces learning, which produces behavior change, which produces results. Research does not consistently support this assumption. Learners can score poorly on satisfaction surveys and still apply what they learned effectively. High Level 2 assessment scores do not reliably predict Level 3 behavior change. The levels are useful measurement categories but they are not reliably causal steps in a chain.
Smile sheet dependency
The ease of collecting Level 1 data creates a risk of treating it as meaningful evaluation. Organizations that report average post-training satisfaction scores as evidence of training effectiveness are misusing the framework. Kirkpatrick explicitly intended Level 1 as the starting point for evaluation, not the end point — but the organizational tendency to stop where data is cheap and available makes this a persistent limitation in practice.
How an LMS Supports Kirkpatrick Model Evaluation
A Learning Management System makes Levels 1 and 2 evaluation operationally straightforward and provides the data infrastructure for Level 3 and 4 analysis.
Level 1 – Reaction: LMS survey tools deploy post-training feedback forms automatically after course completion. Responses are stored, aggregated, and reportable by course, cohort, or time period. This eliminates the manual survey distribution that previously made Level 1 data collection inconsistent. Managers can review team-level satisfaction and relevance scores in reporting dashboards without requesting data exports from the L&D team.
Level 2 – Learning: Assessment scores, quiz pass rates, attempt counts, pre/post score comparisons, and certification completions are all captured and stored automatically in the LMS. Score distributions across the learner population identify which assessment questions most commonly produce incorrect answers pointing directly to the content areas requiring improvement. LMS reporting and analytics make this data available in real time, not just in periodic reports.
Level 3 – Behavior: While the LMS cannot directly measure on-the-job behavior, it supports Level 3 evaluation through follow-up assessment scheduling (automated reassessments at 30, 60, 90 days post-training), manager feedback tools built into the platform, performance review integration where connected to HR systems, and learning pathway tracking that shows which additional modules a learner has accessed since the initial training — a signal of continued engagement with the topic.
Level 4 – Results: An LMS with integration capabilities connects learning activity data to the operational and business performance systems where Level 4 KPIs live CRM systems for sales performance, HRIS for retention and productivity, quality management systems for error rates. This connection makes it possible to correlate training completion and assessment performance data with operational KPI movement across the same employee population.
Frequently Asked Questions
What is the Kirkpatrick Model?
The Kirkpatrick Model is a four-level framework for evaluating training effectiveness. Developed by Donald Kirkpatrick in 1959 and expanded by James Kirkpatrick, it measures training across four levels: Level 1 (Reaction- learner satisfaction and perceived relevance), Level 2 (Learning – knowledge, skills, or attitude acquired), Level 3 (Behavior – on-the-job application of learning), and Level 4 (Results – measurable business impact). The Kirkpatrick model of training evaluation is the most widely used training evaluation framework in corporate L&D globally.
What are the four levels of the Kirkpatrick Model?
The four levels of the Kirkpatrick model are: Level 1 – Reaction (how learners respond to the training experience, measured through post-training surveys); Level 2 – Learning (what knowledge or skills learners gained, measured through assessments, simulations, and pre/post tests); Level 3 – Behavior (whether learners apply learning on the job, measured through manager observation, performance data, and follow-up assessments at 30/60/90 days); and Level 4 — Results (what business impact training produced, measured through operational KPI data compared against the pre-training baseline).
How do you apply the Kirkpatrick Model in corporate training?
Apply the Kirkpatrick evaluation model by starting at Level 4 and working backward: identify the specific business KPI the training is designed to improve and establish a baseline measurement. Define the Level 3 behaviors that would produce that result. Design Level 2 assessments that test whether learners can perform those behaviors. Plan Level 1 surveys focused on relevance and applicability. Then deliver training, collect data at each level, and evaluate whether training moved the Level 4 KPI. Beginning evaluation planning before the course is built — not after delivery — is the critical discipline that distinguishes effective Kirkpatrick implementation from checkbox evaluation.
What is the difference between the Kirkpatrick Model and the Phillips ROI Model?
The Kirkpatrick model measures training effectiveness across four levels (Reaction, Learning, Behavior, Results) without producing a financial return-on-investment calculation. The Phillips ROI Model extends Kirkpatrick by adding a fifth level – converting Level 4 results data into a monetary ROI figure by calculating the financial value of business outcome improvements relative to the total cost of the training program. The Phillips model is more resource-intensive and is typically applied to high-cost programs where financial justification is required. The Kirkpatrick model is more practical for everyday corporate training evaluation where behavior change and operational KPI improvement are the primary evidence of effectiveness.
What is Level 3 of the Kirkpatrick Model and why is it the hardest to measure?
Level 3 of the Kirkpatrick model Behavior measures whether learners apply what they learned in their actual work context after training is complete. It is the hardest level to measure because it requires data collection 30–90 days after training completion, involves collaboration with managers and operational teams outside the L&D function, and cannot be automated by the LMS the way Levels 1 and 2 can. Research indicates that fewer than 15% of corporate training programs conduct consistent Level 3 evaluation meaning most organizations know whether learners passed their courses but not whether they changed how they work as a result.
What is the New World Kirkpatrick Model?
The New World Kirkpatrick Model is a refinement of the original Kirkpatrick framework developed by James Kirkpatrick and Wendy Kayser Kirkpatrick. Its primary contribution is inverting the sequence of planning: rather than designing training at Level 1 and evaluating backward to Level 4, the New World model begins at Level 4 (identifying the business outcome), then defines Level 3 behaviors required to produce that outcome, then designs Level 2 assessment and Level 1 experience to support those behaviors. It also introduces the concept of “required drivers” — the reinforcement, rewards, and accountability systems that must be in place for Level 3 behavior transfer to occur. The framework treats training as one component of a performance improvement system rather than a standalone event.
Can you use the Kirkpatrick Model for eLearning?
Yes — the Kirkpatrick model applies directly to eLearning and is well-supported by LMS technology. Level 1 reaction surveys are automated through LMS course completion workflows. Level 2 assessments are built directly into eLearning courses using authoring tools, with scores stored and reportable in the LMS. Level 3 behavior measurement uses LMS follow-up assessment scheduling combined with manager observation and performance data. Level 4 uses the same KPI comparison methodology as any other training format. eLearning programs are in some respects easier to evaluate at Level 2 than instructor-led training because assessment data is captured consistently for every learner automatically, rather than depending on facilitator-administered assessments with varying quality.
Conclusion
The Kirkpatrick model of training evaluation has remained central to corporate L&D for over 65 years because it asks the question organizations most need answered but most consistently fail to answer rigorously: did the training actually change anything? The four levels provide a structure for moving from the easy data (completion rates, satisfaction scores) to the meaningful data (behavior change, business impact) in a way that connects L&D investment directly to organizational performance.
The practical challenge is not understanding the Kirkpatrick levels most L&D professionals can explain Reaction, Learning, Behavior, and Results clearly. The challenge is organizational discipline: defining Level 4 outcomes before building training, establishing baselines before delivery, planning Level 3 measurement before launch, and following through on data collection 60–90 days after the training has ended and moved off the radar. These are not methodology problems. They are execution problems that require L&D teams to treat evaluation as a core deliverable of every training program not an optional activity to undertake if time allows after the course goes live.
Related resources:
- What Is the ADDIE Model? — instructional design framework that integrates Kirkpatrick evaluation planning
- Training Needs Assessment — establishing the Level 4 outcome before training design begins
- Adult Learning Theory — the science behind designing for Level 3 behavior transfer
- What Is Instructional Design? — connecting evaluation planning to course design

