1. About the Cochrane ATM toolkit
2. Planning a workshop for systematic review authors
3. Implementing a workshop for systematic review authors
4. Evaluating a workshop

Download PDF

1. About the Cochrane ATM toolkit

1.1 Who is the toolkit for?

The toolkit was developed for members of the Cochrane Trainers’ Network that train Cochrane authors in conducting systematic reviews of interventions.

1.2 What is in the toolkit?

The ATM toolkit comprises a trainers’ guide and various learning packages. 

1.2.1 Trainers' guide

The trainers’ guide is intended to be used in conjunction with the learning packages within the ATM toolkit. It provides an overview of the toolkit and a practical guide with tips for trainers on planning, implementing and evaluating workshops¹ for systematic review authors and other groups of learners.

¹The term ‘workshop’ is used throughout as a collective description for any type of training of any duration.

1.2.2 Learning packages

The ATM learning packages are structured according to the steps of a systematic review. The learning packages are currently being updated, and once completed, each learning package will contain downloadable learning material including PowerPoint presentations on theoretical concepts, links to relevant MECIR standards, interactive exercises, multimedia, quizzes, a set of questions for assessment of knowledge, and package-specific guidance. 
Currently, the ATM toolkit is only available in English. 

1.3 What is the purpose and the scope of the tookit?

The aim of the ATM toolkit is to provide Cochrane trainers with comprehensive guidance and a variety of learning materials needed to successfully facilitate a workshop¹ for Cochrane authors. The toolkit allows trainers to tailor their workshops according to the target audience, the context within which it is being offered and the available time.

1.3.1 What can I expect from the toolkit?

• It provides Cochrane trainers with the learning materials needed to facilitate training on Cochrane systematic reviews of interventions
• The content covers all basic concepts that authors need to understand to be able to conduct a systematic review of interventions following Cochrane methods
• The learning material has primarily been developed for use in face-to-face workshops rather than online learning
• It offers Cochrane trainers the flexibility to customize the format of each session according to their unique workshop
• It provides guidance on planning, implementing and evaluating a workshop

1.3.2 What can I not expect from the toolkit?

• The content does not cover specialized topics such as diagnostic test accuracy reviews, qualitative evidence syntheses, individual patient data meta-analysis or network meta-analysis
• There are no prescribed learning outcomes, as these depend on the nature of the workshop and should be developed by Cochrane trainers (see section 2.2)  
• There is no prescribed format for sessions

1.4 Where can I access the tookit?

The toolkit is available on the landing page of the Trainers' Hub.

1.5 Who developed the toolkit?

The content of each learning package was developed by experts from the relevant Cochrane Methods Groups. Acknowledgements are provided in the final slide of Power Point presentations in each learning package.
The format of the ATM toolkit was developed, and user interface built by Dario Sambunjak and Chris Watts (Learning team, Cochrane’s People Services Department).
This trainers guide, as well as the specific guides available in individual learning packages, have been developed by Anke Rohwer (Centre for Evidence-based Health Care, Division Epidemiology and Biostatistics, Faculty of Medicine and Health Sciences, Stellenbosch University) in collaboration with Dario Sambunjak and Chris Watts.
Updating of the materials in this toolkit was done in 2021 by Anke Rohwer and Dario Sambunjak. Other contributors and content reviewers were: Stephanie Boughton, Samantha Cox, Liz Dooley, Robin Featherstone, Ursula Gonthier, Rebecka Hall, Anupa Shah, Jana Stojanova and Areti Angeliki (Argie) Veroniki.

Back to top

2. Planning a workshop for systematic review authors

Adequate and careful planning will pave the way for a successful workshop. Before you assemble your session using the learning packages in the toolkit, you need to consider the target audience of your workshop, the aim of your workshop, the learning outcomes and how you are going to achieve these outcomes. Below we give some guidance on these aspects and use three scenarios to illustrate this. 

2.1 Who is the target audience?

It is essential to know who your audience will be, as you will tailor your workshop accordingly. Some characteristics to consider include:

• Experience with conducting systematic reviews
• Background knowledge/previous training
• Professional background
• Context of participants (includes country where workshop is being held)
• Native vs. non-native English speakers
• Expected number of participants

2.2 What do you want to achieve during the workshop?

This is probably the most important step in planning your workshop, as it will guide you in selecting relevant learning packages, selecting relevant learning material from each learning package and assembling a session, or various sessions for your workshop.

2.2.1 Aim of the workshop

The aim of the workshop is a broad statement of what you want to achieve during the workshop. It indicates the general content and direction of the training and is written from the trainer’s perspective.

2.2.2 Learning outcomes

Learning outcomes are statements of what a learner is expected to know, understand or be able to demonstrate after completion of the training². Developing clear and meaningful learning outcomes is important for you as the trainer, as it will help you to plan what to cover, to what extent to cover it and which learning material and activities to use to achieve the outcomes. Clear learning outcomes are equally important for the workshop participants, as they will know what to expect from the workshop in terms of the knowledge and skills they will gain from the workshop.

²ECTS Users' Guide, 2005

To allow flexibility of the format of workshops for Cochrane authors, the learning packages in the toolkit do not contain generic learning outcomes. Cochrane trainers are therefore encouraged to develop learning outcomes that are specific to their workshop. In some cases, it might be easier to develop learning outcomes for each session, but this depends on the scope and the duration of the workshop.

2.2.2.1 How do I develop learning outcomes?

Bloom’s taxonomy³, a framework of moving from simple to more complex levels of understanding, is a useful tool when developing learning outcomes for your workshop, as it helps you to think through the level of understanding that is expected from participants. The original taxonomy was created by Benjamin Bloom in 1956 and was revised in 2001. It includes six levels of learning, each level building on the previous one (see Figure 1).

Figure 1: Bloom's taxonomy of learning⁴

³David R. Krathwohl (2002): A Revision of Bloom's Taxonomy: An Overview, Theory Into Practice, 41:4, 212-218
⁴Reproduced under the Creative Commons Attribution licence from Vanderbilt University Center for Teaching (https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/)

Tips for writing learning outcomes:

• Begin each outcome with an action verb
• Use only one verb per outcome
• Verbs should be matched to the level of understanding expected (see Table 1)
• Don’t use complicated sentences
• Outcomes should be clear and measurable
• Avoid vague terms such as know or understand
• Keep in mind available time
• Keep in mind level of participants

Table 1: Suggested verbs for learning outcomes according to Bloom’s taxonomy

2.2.3 How do I apply this to my workshop?

First, you need to be clear about the aim of the workshop. Learning outcomes will depend on the aim of the workshop, the target audience and the time available. Think about what it is that you want participants to be able to do at the end of the workshop. Below are a few examples that can help you think through potential aims and learning outcomes for your workshop.

Scenario 1: You have been asked to offer training to clinicians from a low-and middle-income setting, that do not have any experience in conducting systematic reviews and have not attended any previous training. You are planning weekly lunch time sessions of 1 hour each with the aim of introducing Cochrane Reviews. The topic of one of your sessions is ‘Phrasing questions for systematic reviews’. You develop the following aim and learning outcomes:

Aim: To introduce the concept of formulating questions within a Cochrane Review

Learning outcomes: After the session, participants will be able to:

• Explain the importance of a clearly phrased question,
• Describe the use of the Population (P), Intervention (I), Comparison © and Outcome (O) framework,
• Formulate a systematic review question using the PICO question.

Scenario 2: You are preparing a session on using the new risk of bias tool (RoB 2) to assess risk of bias of randomized controlled trials (RCT) for Cochrane authors that have all previously conducted Cochrane Reviews. You are planning a 2-hour face-to-face session but expect participants to do some self-study before they attend the session. You develop the following aim and learning outcomes:

Aim: To provide insight into using the new RoB 2 tool for Cochrane Reviews

Learning outcomes: After the session, participants will be able to:

• Explain the domains of the RoB 2 tool,
• Evaluate risk of bias of a RCT using the RoB 2 tool,
• Justify judgements of risk of bias,
• Compare judgements of risk of bias with others and resolve discrepancies.

Scenario 3: You are planning to offer an introductory workshop to PhD students that are interested in conducting systematic reviews as part of their studies. The workshop is divided into two 5-day blocks. The first week covered phrasing questions, searching, extracting data and assessing risk of bias. You develop the following overarching aim and learning outcomes for the second week:

Aim: To explain the process of analyzing and synthesizing data within a systematic review and to provide an overview of grading the quality of evidence

Learning outcomes: After the workshop, participants will be able to:

• Analyze dichotomous and continuous outcomes,
• Explain the value and process of meta-analysis,
• Interpret forest plots,
• Define heterogeneity,
• Discuss how to recognize and deal with heterogeneity,
• Describe reporting bias,
• Explain certainty of evidence and the GRADE approach,
• Apply GRADE to assess certainty of evidence.

2.3 How will you ensure that participants achieve the learning outcomes?

Once you have developed clear learning outcomes for your workshop, you need to consider how you will ensure that participants achieve the learning outcomes. Everything that you do during the workshop, including didactic input and interactive exercises, should therefore be aligned with the aim of the workshop, the learning outcomes and the level of expected understanding as described in section 2.2. The next step is therefore to select the relevant learning package from the ATM toolkit, consider which learning material to use, and assemble each session of your workshop.

2.3.1 Selecting relevant learning material from the learning packages

Learning packages in this toolkit contain presentations on theoretical concepts, links to MECIR standards, interactive exercises, quizzes, multimedia, a set of questions for assessment of knowledge, and a package-specific guidance. Although you can use all the available material, you do not necessarily have to use all of it. Indeed, this is where you can choose what is most relevant to your context, your audience and the available time. Keep in mind, that you can share some of the materials before or after the workshop. Your learning outcomes and the verbs used as part of these, will help you to determine which materials will be most useful.

Below we illustrate this using the three scenarios described previously.

In Scenario 1, you are planning a 1-hour session on ‘Phrasing questions for systematic reviews’ for clinicians. Your learning outcomes state that, after the session, participants will be able to explain the importance of a clearly phrased question; describe the use of the PICO framework; and formulate a systematic review question using the PICO framework. First, you consider the verbs that indicate the level of understanding that is expected. The verbs used in the first two learning outcomes, explain and describe, refer to the level of ‘understanding’. This means that participants will learn new facts or concepts, in this case, the importance of phrasing clear questions and the use of the PICO framework. Introducing new knowledge is best done using the PowerPoint presentation on the theoretical concepts in the learning package on phrasing questions. The third learning outcome states that participants will formulate a question. This verb indicates that participants will need to do something, and it refers to the level of ‘creating’ something new. You therefore need an activity or a suitable exercise that participants can do during the session. As you will have limited time during this 1-hour session, you therefore only select the PowerPoint presentation and the exercise on phrasing questions from the learning package.

In Scenario 2, you are planning a workshop on the new RoB 2 tool for Cochrane Review authors. Your learning outcomes state that after the workshop, participants will be able to explain the domains of the RoB 2 tool, evaluate risk of bias of a RCT using the RoB 2 tool, justify judgements of risk of bias, compare judgements of risk of bias with others, and resolve discrepancies. The first learning outcome indicates that participants need to ‘understand’ a new concept, while the rest of the learning outcomes indicate that participants need to do something, namely ‘apply’ the RoB 2 tool and ‘evaluate’ risk of bias in a RCT. As per the previous scenario, you select the PowerPoint presentation (or a recorded webinar) to introduce new knowledge, as well as the risk of bias exercise in the learning package (DROPLET exercise). As your participants are Cochrane authors, you also refer participants to the MECIR standards as this will be important when conducting a Cochrane Review.

In Scenario 3, you are planning the second week of a workshop to introduce systematic reviews. You developed learning outcomes for the whole week and now need to plan the individual sessions. The first session is on analyzing dichotomous data. You start by developing the following more detailed learning outcomes for this session: After this session, participants will be able to explain dichotomous outcomes and how they are measured; give examples of the measures if effects used to compare two groups for a dichotomous outcome; interpret risk ratios and odds ratios, calculate risk ratios and odds ratios, discuss the value of 95% confidence intervals (CIs) and interpret 95%CI. Most of the learning outcomes for this session relate to ‘understanding’ of new concepts. You therefore select the PowerPoint presentation that explains the theoretical concepts and check whether there is any relevant multimedia that can enhance or reinforce the content of the presentation. As you did include learning outcomes that refer to ‘applying’ the new knowledge, you also select the exercise on dichotomous outcomes that is provided in the learning package. You believe that it is important that participants can correctly interpret risk ratios, odds ratios and 95%CIs for them to understand meta-analysis (which you are planning to cover in a subsequent session), you also select the set of quiz questions from the learning package that you aim to integrate into the presentation.

2.3.2 Assembling a session

Now that you have selected the relevant material from the learning packages, you can assemble your session. Keep in mind the following when assembling a session:

• Aim to mix didactic input with interactivity to keep energy levels up
• Consider the flow of the session, e.g. introduce a new concept before interactive exercises
• Decide whether interactive exercises will be individual or small group exercises
• Consider the available time in terms of the duration of the workshop, the time available per session or day, time needed for didactic input and exercises
• Allow enough time for interactive exercises
• Plan enough time for participants to talk to each other
• Include enough breaks (tea, lunch, comfort or stretch)
• Always include time for feedback, discussions and questions

Below we provide examples of how to assemble sessions linked to the three scenarios described previously.

Scenario 1: Phrasing questions (1 hour session)

• PowerPoint presentation: 15 minutes
• Formulating PICO questions exercise: 20 minutes
• Feedback and discussion: 20 minutes
• Wrap up: 5 minutes

Scenario 2: RoB 2 tool (2-hour session with self-study before the workshop)

• Recorded webinar on the RoB 2 tool: participants to watch before the workshop
• Summary of RoB 2 tool (PowerPoint presentation) and clarifying questions: 30 minutes
• Individual exercise: reading RCT and assessing risk of bias using RoB 2 tool: 40 minutes
• Small group discussion comparing risk of bias judgements and resolving discrepancies: 30 minutes
• Feedback and discussion in big group: 20 minutes

Scenario 3: Analyzing dichotomous data (morning session 1.5 hours)

• PowerPoint presentation and integrated quiz: 60 minutes
• Small group exercise on calculating risk ratios and odds ratios: 20 minutes
• Feedback and discussion: 10 minutes

2.4 Who will facilitate the workshop?

When planning a workshop, it is useful to involve a team of facilitators at an early stage. Facilitators can have various roles, with some, usually the more experienced facilitators, presenting didactic content and others helping with small group exercises. Having a team of facilitators as opposed to offering the workshop on your own has various advantages. You can include facilitators with expertise in various areas, so that you will be able to complement each other when facilitating a workshop. This will enable you to share the workload, have more people to help participants during hands-on exercises, and reflect on what went well and what did not go well with peers. Furthermore, if you include a mix of novice and experienced facilitators, you can build capacity of trainers while offering the workshop. The number of facilitators needed depends on the number of sessions being offered, the number of participants expected and availability of facilitators.

2.5 When and where will the workshop be offered?

Planning the logistics of a workshop is just as important as planning the content of the sessions, although trainers are not always responsible for organizing the logistics of the workshop. If you are responsible for the logistics, make sure you consider the following:

• Propose a date and time of the workshop as soon as possible
• Find a suitable venue for the workshop
• Determine the maximum number of participants that you can accommodate
• Check whether the venue is available during the proposed time before you confirm the date and venue
• Consider the set-up of the venue – for a workshop, a few round tables with chairs is preferred over a lecture-style set-up
• Check whether participants will have access to Wi-Fi
• Make sure that the venue has all the equipment that you will need such as laptops, a sound system, a projector, a white board etc.
• Consider the ambiance of the venue - lighting, air conditioning, windows
• Make sure there is enough space for activities
• Organize refreshments, water and meals if needed
• Consider whether some participants or facilitators might need accommodation and suggest appropriate options

2.6 How will you advertise the workshop?

Once you have confirmed the date and venue for your workshop, you will be able to advertise the workshop. There are various ways that you can recruit participants ranging from a general advert on an online platform, to a personal email as part of a more targeted strategy. Importantly you need to think about the best way to reach your target audience.

Below are a few tips on what to include in an advert:

• Name of the workshop
• Date, time and venue
• Aim and learning outcomes of the workshop
• Who the workshop is for
• How to register for the workshop and deadline for registrations
• Costs involved
• Contact details

2.7 How will you evaluate the workshop?

You need to plan how you will evaluate your workshop at the planning stage. More detailed guidance on evaluating a workshop is provided in section 4 of this guide.

2.8 How will you put everything together?

Once you have planned your workshop by considering everything that we mentioned above, we recommend that you develop a workshop plan that contains the outline of each session, the learning outcomes, the resources and handouts needed, as well as the name of the facilitator that will be responsible for the session. In essence, the workshop plan will become your roadmap for the entire workshop.

Below is an example of a workshop plan for the first scenario described above.

Scenario 1

2.9 Further resources linked to planning a workshop

Developing learning outcomes

• https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy
• https://tips.uark.edu/using-blooms-taxonomy
• https://www.utica.edu/academic/Assessment/new/Blooms%20Taxonomy%20-%20Best.pdf  

Back to top

3. Implementing a workshop for systematic review authors

If you are a seasoned Cochrane trainer, you most probably have a lot of experience in implementing workshops for systematic review authors. If you are a novice trainer, you might be less familiar with the learning materials and facilitating a workshop might seem a bit daunting. We therefore provide a few tips on using the learning materials and on effective facilitation.

3.1 Using the learning materials in the learning packages

In the previous section, we provided guidance on choosing learning material and assembling a session. Here, we will be focusing on using those materials across learning packages. More guidance specific to each learning package is provided in the package-specific guidance.

Using the PowerPoint presentations

The PowerPoint presentations in the learning packages have been carefully developed by experts from the Cochrane Methods group. The content of the slides should, in principle, not be changed.

However, depending on your target audience, you need to check whether all the content is relevant. For example, if you are preparing an introductory workshop, as in Scenario 1, you will only need to cover the basic content on phrasing questions. Content on more complex questions for systematic reviews will not be applicable in this context and you can thus omit this. Furthermore, you have only planned 20 minutes for the presentation during the workshop. You therefore need to check that you can present all the necessary content within this timeframe. Don’t rush the presentation, rather include less slides and have enough time for the presentation.

If you are planning to integrate discussion points, multi-media or a quiz into the presentation, as is the case in Scenario 3, adding a blank slide at the correct point in the presentation can act as a reminder or a placeholder to initiate the quiz.

All slides contain detailed notes for the trainer. The intention is not that you read the notes while presenting, but rather that you familiarize yourself with the content of the slides before the workshop so that you can present them comfortably. In addition, the notes act as a reminder of the key points during the presentation. Practice your presentation and make sure that you know the slides before the workshop. Try to use the slides to emphasise what you want to say, and don’t read your slides or talk to your slides – rather talk to your audience.

Interactive exercises

Interactive exercises are a key part of any workshop as active participation in learning will enable meaningful and deep learning. Exercises or discussions in small groups is a great way to facilitate engagement between participants and enhance collaborative learning.

All learning packages contain detailed guidance on facilitating the specific exercises included as part of the learning package. Make sure that you are familiar with the exercise that you are planning to incorporate into your workshop. Think about your audience and what would be appropriate and relevant. For example, in Scenario 1, you have planned a small group exercise on phrasing questions. As your participants will be clinicians from a low-and middle-income setting, it will be most relevant to use the example linked to hand-hygiene, as this is likely to be more relevant to their setting than the example on coronary heart disease and will thus make the learning experience more authentic. In Scenario 2, your participants will be experienced Cochrane authors and you are planning to use the exercise on critical appraisal. Even though the learning package contains a trainers’ version of the exercise containing the answers, it is always good to read the RCT and assess the risk of bias before the workshop, so that you are familiar with the content.

If you want to divide participants into small groups, think about how you are going to do this before the workshop, as this can take up a lot of time. Think about fun and innovative ways to allocate participants to groups that involves them getting up from their seats. Not only will this be an opportunity for participants to get up and move around (equivalent to an energizer or stretch break), but it will also allow them to engage with participants that are not seated at their table.

Adequate preparation is essential for interactive exercises to go smoothly. You need to make sure that you have all the resources you need, including worksheets, flipcharts, marker pens or anything else that you might need. If you are planning to incorporate an online quiz, as in Scenario 3, set up the quiz using software you are familiar with and check that it is working on the day of the workshop. Most importantly, make sure that participants will have access to Wi-Fi so that they can participate in the quiz.

During the workshop, make sure that participants understand what the exercise entails and what is expected of them. In addition to a clear explanation of the activity, you can add key points of the exercise on a slide or on a flipchart and keep this up during the exercise. Remember to include everything that participants need to do – if they are working in small groups, as in Scenario 1, they will need to discuss the problem, identify the PICO components and formulate the question. They need to write the PICO elements and the question on the flipchart and select one person from the group to give feedback.

Be available to answer any questions that come up during the exercises. This is where you will engage the whole team of facilitators to move around the room and interact with participants and small groups. Remember that you sometimes need to prompt participants – they will not always feel comfortable to ask questions.

Integrating quiz questions into your presentation

You can assess whether participants have achieved the learning outcomes that you put forward in many ways and at different time points. Incorporating self-assessment into the workshop is a great way for learners and facilitators to gauge whether participants have understood the content. Furthermore, you will know which concepts need clarification or further explanations and examples.

Some learning packages contain a set of quiz questions, which can be used to assess learning during and after a session. The quiz questions are provided as a set of PowerPoint slides, and you can use interactive presentation software or non-technological methods to deliver the quiz. Interactive software such as Mentimeter or Poll everywhere allows participants to answer quiz questions in real time, using their mobile phones or laptops. However, you need a stable internet connection and access to Wi-Fi for all participants. Non-technological methods include color-coding potential answers of the quiz and handing out a set of the same color-coded cards to all participants. Participants can then answer a question by holding up the correctly colored card. Another very simple method for participants to answer quiz questions is to raise their hands.

You can watch the following videos for information on using Mentimeter software:

• Beginners guide to Mentimeter
• Mentimeter for the digital classroom
• https://www.mentimeter.com/webinars

Multimedia

Some learning packages contain links to useful videos. These are mostly short clips, explaining a certain aspect linked to the learning package. For example, the learning package ‘Dichotomous outcomes’ contains a short video clip on calculating risk ratios, odds ratios and risk differences. You can incorporate these in your workshop by sharing the links with participants so that they can watch them in their own time, or you play the video during the workshop to supplement the PowerPoint presentations. If you are planning to play the video during the workshop, make sure that the venue is equipped with suitable audio equipment. We suggest that you familiarize yourself with the content of the videos before sharing them with participants.

Other resources

Learning packages also contain other useful resources for participants and trainers to complement the training offered during the workshop. For example, each learning package contains the link to the relevant Cochrane Interactive Learning module. Trainers can work through the relevant modules before the workshop, to familiarize themselves with the topic, if needed. This is particularly useful for more novice trainers. You can also ask participants to work through the online modules (for access details, see subscription options to Cochrane Interactive Learning), or watch a recorded webinar before the workshop, so that you can spend more time on interactive exercises, as proposed for scenario 2 in section 2.3.2.

3.2 Tips for facilitating a workshop

Come prepared

• Good planning and preparation is key
• Come to the venue early so that you have enough time to settle in and set up
• Test the technology to make sure everything is working
• Set up the venue or check that it is set up correctly
• Prepare handouts, flip charts, marker pens or any other resources
• Ensure enough fresh air in the room

Know your audience

• Always start with a round of introductions but keep it short
• Know the background of participants
• Ask about the expectations of participants

Create rapport

• Be authentic and share your personal experiences with reviews
• Be honest
• Be approachable – there is no such thing as a dumb question
• Talk to participants during the tea and lunch breaks

Be in-tune with your audience

• Observe your audience and listen actively to what they are saying
• Check understanding of concepts
• Check and maintain energy levels of participants

Communicate clearly

• Make sure that all participants can hear you
• Don’t talk to fast
• Don’t use fancy words – make sure everyone understands what you are saying
• Make eye contact with participants
• Be aware of your body language

Manage time

• Make sure participants know the schedule for the day and keep to the schedule as far as possible, but be flexible to allow for longer/shorter discussions or exercises
• If more time is needed e.g. for an exercise or a discussion, check-in with audience on how to reschedule activities for the rest of the day
• Start and end on time

Keep participants engaged

• Stimulate discussions between participants
• Manage group dynamics
• Ask questions and wait for an answer
• Respond to questions
• Redirect questions to participants
• Include relevant examples

Support participants during exercises

• Move around the room
• Actively ask groups/individuals how the exercise is going
• Make sure there are enough facilitators to help with exercises
• Help those who are struggling

Be prepared for unexpected changes

• Be aware that a lot of things can change unexpectedly
• Be responsive and able to adapt the plan
• Learn to think on your feet
• Know who to ask for help e.g. in case of technical issues

Have fun

• Include fun activities such as ice breakers and energizers
• Use innovative ways to allocate participants to groups
• Don’t be too serious

Back to top

4. Evaluating a workshop

It is important to evaluate your workshop to determine its effectiveness and value; and to determine whether your approach was useful in achieving the learning outcomes. This will help you to improve subsequent workshops. Evaluating your workshop can include feedback from participants, feedback from peers, and your own reflections.

4.1 Kirkpatrick Model of Evaluating Learning

The Kirkpatrick model is a useful and well-known framework to evaluate learning⁵. It consists of four levels: 1) Reaction, 2) Learning, 3) Behaviour and 4) Results. While it will be difficult to evaluate your workshop on all these levels, it is usually feasible to evaluate Reaction and Learning.

⁵https://www.mindtools.com/pages/article/kirkpatrick.htm

4.1.1 Level 1: Reaction

Evaluating how participants react to your training gives you useful information on how participants perceived the training. The best and easiest way of obtaining feedback from participants is through a questionnaire.

Developing the questionnaire

• Develop your questionnaire before the workshop (at planning stage)
• Try to stick to a few questions that will provide actionable responses
• Include Likert-scale questions and/or open-ended questions
• Questions can cover the content, the presentation style, learning activities and suggestions for improvement
• Examples of Likert-scale questions
  - Did the presentations meet your expectations?
  Very dissatisfied, dissatisfied, neutral, satisfied, very satisfied
  - How would you rate the quality of presentations?
  Very poor, poor, average, good, excellent
  - Were the class exercises useful to enhance learning?
  Not at all useful, not useful, neutral, useful, very useful
  - I would recommend this workshop to other Cochrane authors.
  Strongly disagree, disagree, neutral, agree, strongly agree
• Examples of open-ended questions
  - What I liked best about the workshop was…
  - What I liked least about the workshop was…
  - My overall impression of the workshop was…
  - What will you do with the information gained during the workshop?
  - Do you have suggestions on how to improve the workshop?

Administering the questionnaire

• Think about whether you want a paper-based or an online questionnaire
• For a paper-based questionnaire
  - Ensure that each participant receives a copy
  - Allocate a few minutes at the end of the workshop for participants to complete the questionnaire
  - Have a box ready where participants can deposit their completed form to ensure anonymity
  - Have spare pens ready
• For an online questionnaire
  - Set up an online questionnaire that does not require participants to identify themselves to ensure anonymity
  - Make sure participants can access the link to the questionnaire
  - Add the link to your last PowerPoint slide, write it on a flipchart, or email it to participants
• Encourage participants to provide feedback

Analyzing feedback

• Don’t take negative feedback personally
• Remember that feedback is about improving the learning experience
• Analyze the feedback critically
• Think about whether it is realistic to include modifications in future workshops
• Discuss difficult feedback with a co-facilitator or a peer

4.1.2 Level 2: Learning

Evaluating learning refers to the knowledge and skills that participants have gained during the workshop. Although you will not be able to measure this in a comprehensive way, we have included knowledge assessments for each learning package that will help both trainers and participants in gauging whether the pre-specified learning outcomes were met. Knowledge assessments consist of multiple-choice questions that are directly linked to the didactic content included in the PowerPoint presentations and can be used in various ways. For example, you can distribute a paper-based version of the assessment at the end of a workshop and ask participants to complete these before they leave; or you can set up an online version of the set of questions and send the link to participants after the workshop. For each learning package, we have included a Trainers’ version of the knowledge assessment which contains the correct answers to the questions.

Some workshops might be targeted specifically at authors that are currently busy conducting a Cochrane Review. In that case, we suggest that you follow-up with participants after the workshop to ask about progress and any outstanding questions.

4.2 Obtaining feedback from peers

It is really useful to obtain feedback from co-facilitators or peers, as these are usually people that you can trust. In addition, they can experience a workshop from the perspective of a participant and facilitator and are thus in a good position to give constructive feedback. One way of doing this is for the team of facilitators to debrief after a workshop. If the workshop runs over a few days, debriefing at the end of each day will help you to make necessary changes during subsequent days.

4.3 Reflecting on the content and processes

Reflecting involves thinking critically about what you do or what you did. Reflection can happen during the workshop, where you as the facilitator need to recognize the need to adapt, change gears or spend more time on a certain topic. Reflection can also happen after the workshop, where you as the facilitator look back on what happened during the workshop, what worked well and what did not work so well. This might include the content as well as the learning strategies and exercises you used. Writing down a few notes will help you to remember what you reflected on when you plan your next workshop.

Back to top