case study program implementation

Via Giacomo Saudino, 36

10015 Ivrea, Italy

[email protected]

(+39) 334 855 0474.   |    [email protected]

The Implementation: Another Crucial, Ingredient for Case Study Success

by John Cole | Jun 21, 2021 | B2B Copywriting , Collateral , Lead Generation , Case Studies , Technology Marketing | 0 comments

When prospects talk with your salespeople about purchasing your solutions, what are some of their biggest concerns?

If you’re like most tech industry executives, I’ll bet your list includes:

• What will it take to implement your solution?
• How long will it take to get it up and running?
• What problems might we run into during the implementation?

Let’s face it. Unless your solution is extremely simple, there’s always an implementation phase. Tech buyers know that until they’ve completed that phase, there will be no ROI. And if the implementation doesn’t go well, there will be schedule delays, unplanned expenses, and lower returns.

So, your prospects want reassurance that implementation will be accomplished quickly and efficiently.

How can you help your sales and technical support staff address those concerns?

Well, if you publish case studies, you have the perfect vehicle. Once you’ve introduced your solution in the story, all you have to do is describe the implementation and how your company supported it. You’ll be supplying your sales reps with solid evidence—from real customers—that you’ve dealt with implementation problems successfully.

Strangely, many case studies fail to sufficiently cover this critical phase. We’ll examine why in a moment. Then, we’ll talk about how to make the most of your implementation discussion. But first, let’s look at why you should address the implementation process in your case studies.

Why the implementation matters in a case study

Like the customer Journey we discussed last time , the Implementation is important to your case study in two ways: first as a marketing force, and second as a structural element for creating drama—for making your customer’s story compelling.

From the marketing viewpoint, consider your audience. Tech buyers are a skeptical bunch. They know that no new system implementation comes off without a hitch. They expect unexpected obstacles. Any marketing claims of “easy installation and set-up” they take with a big grain of salt.

So, in your case studies, it’s important to describe the implementation in detail.

You need to talk about any difficulties or obstacles that were encountered and how they were overcome. This builds credibility with tech buyers. It answers some of their most important questions. It shows them you can handle those challenges. And it makes the results you describe later more believable.

It may seem counter-intuitive, but revealing the challenges in the implementation process—and how they were successfully resolved, of course—makes both your solution and your story more compelling for your reader.

The storytelling importance of the implementation

Which leads us to the storytelling importance of the implementation…

One of the most important forces in any story is the Resolution of Conflict, which begins when the hero of the story—just when all seems lost—determines exactly what she must do. In a case study, that’s where the customer discovers your solution. The implementation, then, is the next logical step. It shows our hero putting her plan into action. The implementation drives the story to its climax.

If you jump too quickly from purchase to results without spending enough time on the implementation, your story will feel hollow. Empty. Your results will seem too good to be true. Your readers need that resolution of the conflict so that they can believe in your “happily ever after.”

Why the implementation gets short-changed

So why does the implementation phase so often get short shrift in case studies? I think there are three reasons.

The most likely reason is that the Implementation, like the Journey, is not one of the four major divisions—i.e., the Customer, the Challenge, the Solution, and the Results—that we usually think of as the basic case study structure. Seasoned case study writers will usually discuss the Implementation as part of the Solution section. But other copywriters—like those from ad agencies, for example, who are trained to emphasize benefits above all else—may focus solely on the solution itself and not appreciate the importance of the Implementation.

A second possible reason is customer apprehension. Some customers may want to suppress or limit the implementation discussion because they fear they may lose a competitive advantage they’ve just gained. Others may feel they may somehow tarnish their corporate image if they reveal they needed help to implement your solution.

A third reason may lie with the solution provider—the company publishing the case study. If problems were encountered during implementation, they may fear it will reflect badly on their solution or their customer, They may, therefore, decide to suppress the implementation discussion.

Obstacles are Opportunities

But if you look at it from your prospect’s perspective, implementation challenges should be embraced in a case study.

Prospects want to know what they’ll be up against. They want to know what it took to successfully apply your solution to the customer’s application. They also want to know how your company helped the customer overcome any problems they encountered, and what the customer thought of the help they received. In other words, implementation problems are opportunities. You can easily turn them into credibility builders!

And when you think about it, problems encountered during implementation don’t really reflect badly on your customer at all. They were just going through what we all go through, dealing with everyday problems as they arise. What counts is how you dealt with those problems.

After all, this is a “success story” we’re talking about. If you hadn’t successfully resolved those issues, you wouldn’t be publishing it, would you? All you really have to do is emphasize the positive outcome: put your customer, company, and solution in the best possible light.

So give your audience a full, blow-by-blow description of the implementation process. Use as much detail as you can in the space available. Detail is the key to credibility. A detailed, engaging Implementation discussion in your case study will make the Results section that follows it all that more believable.

Take-Away Points

1. Case study readers (your prospects) want to know exactly:

a. What was required to implement your solution

b. How your company helped your customer overcome any obstacles encountered

2. Every case study should include a detailed, description of the system implementation.

3. Revealing challenges in the implementation process actually builds credibility and makes both your story and your product more compelling to technology buyers.

4. It’s not the problems but how you dealt with them that matters.

Need help crafting a compelling case study – one that includes a description of the implementation and follows the rules of good storytelling? Contact CopyEngineer by email at [email protected] .

Contact CopyEngineer

(See our  privacy policy )

OR Click here for more information.

Free Report

Not ready to talk about a new copywriting project just yet contact me anyway to get your free copy of my latest special report:.

10 Common Mistakes That Kill White Paper ROI How to Avoid Them and Generate More Leads

And don’t forget to sign up for my FREE e-zine,  Technical Response . Not only will you receive tips for better content marketing and lead generation. You’ll also receive a copy of my white paper, How to Plan a White Paper: A Proven 7-Step Process for Minimizing Headaches and Maximizing ROI , absolutely FREE. Subscribe now!

How to write a case study — examples, templates, and tools

It’s a marketer’s job to communicate the effectiveness of a product or service to potential and current customers to convince them to buy and keep business moving. One of the best methods for doing this is to share success stories that are relatable to prospects and customers based on their pain points, experiences, and overall needs.

That’s where case studies come in. Case studies are an essential part of a content marketing plan. These in-depth stories of customer experiences are some of the most effective at demonstrating the value of a product or service. Yet many marketers don’t use them, whether because of their regimented formats or the process of customer involvement and approval.

A case study is a powerful tool for showcasing your hard work and the success your customer achieved. But writing a great case study can be difficult if you’ve never done it before or if it’s been a while. This guide will show you how to write an effective case study and provide real-world examples and templates that will keep readers engaged and support your business.

In this article, you’ll learn:

What is a case study?

How to write a case study, case study templates, case study examples, case study tools.

A case study is the detailed story of a customer’s experience with a product or service that demonstrates their success and often includes measurable outcomes. Case studies are used in a range of fields and for various reasons, from business to academic research. They’re especially impactful in marketing as brands work to convince and convert consumers with relatable, real-world stories of actual customer experiences.

The best case studies tell the story of a customer’s success, including the steps they took, the results they achieved, and the support they received from a brand along the way. To write a great case study, you need to:

• Celebrate the customer and make them — not a product or service — the star of the story.
• Craft the story with specific audiences or target segments in mind so that the story of one customer will be viewed as relatable and actionable for another customer.
• Write copy that is easy to read and engaging so that readers will gain the insights and messages intended.
• Follow a standardized format that includes all of the essentials a potential customer would find interesting and useful.
• Support all of the claims for success made in the story with data in the forms of hard numbers and customer statements.

Case studies are a type of review but more in depth, aiming to show — rather than just tell — the positive experiences that customers have with a brand. Notably, 89% of consumers read reviews before deciding to buy, and 79% view case study content as part of their purchasing process. When it comes to B2B sales, 52% of buyers rank case studies as an important part of their evaluation process.

Telling a brand story through the experience of a tried-and-true customer matters. The story is relatable to potential new customers as they imagine themselves in the shoes of the company or individual featured in the case study. Showcasing previous customers can help new ones see themselves engaging with your brand in the ways that are most meaningful to them.

Besides sharing the perspective of another customer, case studies stand out from other content marketing forms because they are based on evidence. Whether pulling from client testimonials or data-driven results, case studies tend to have more impact on new business because the story contains information that is both objective (data) and subjective (customer experience) — and the brand doesn’t sound too self-promotional.

Case studies are unique in that there’s a fairly standardized format for telling a customer’s story. But that doesn’t mean there isn’t room for creativity. It’s all about making sure that teams are clear on the goals for the case study — along with strategies for supporting content and channels — and understanding how the story fits within the framework of the company’s overall marketing goals.

Here are the basic steps to writing a good case study.

1. Identify your goal

Start by defining exactly who your case study will be designed to help. Case studies are about specific instances where a company works with a customer to achieve a goal. Identify which customers are likely to have these goals, as well as other needs the story should cover to appeal to them.

The answer is often found in one of the buyer personas that have been constructed as part of your larger marketing strategy. This can include anything from new leads generated by the marketing team to long-term customers that are being pressed for cross-sell opportunities. In all of these cases, demonstrating value through a relatable customer success story can be part of the solution to conversion.

2. Choose your client or subject

Who you highlight matters. Case studies tie brands together that might otherwise not cross paths. A writer will want to ensure that the highlighted customer aligns with their own company’s brand identity and offerings. Look for a customer with positive name recognition who has had great success with a product or service and is willing to be an advocate.

The client should also match up with the identified target audience. Whichever company or individual is selected should be a reflection of other potential customers who can see themselves in similar circumstances, having the same problems and possible solutions.

Some of the most compelling case studies feature customers who:

• Switch from one product or service to another while naming competitors that missed the mark.
• Experience measurable results that are relatable to others in a specific industry.
• Represent well-known brands and recognizable names that are likely to compel action.
• Advocate for a product or service as a champion and are well-versed in its advantages.

Whoever or whatever customer is selected, marketers must ensure they have the permission of the company involved before getting started. Some brands have strict review and approval procedures for any official marketing or promotional materials that include their name. Acquiring those approvals in advance will prevent any miscommunication or wasted effort if there is an issue with their legal or compliance teams.

3. Conduct research and compile data

Substantiating the claims made in a case study — either by the marketing team or customers themselves — adds validity to the story. To do this, include data and feedback from the client that defines what success looks like. This can be anything from demonstrating return on investment (ROI) to a specific metric the customer was striving to improve. Case studies should prove how an outcome was achieved and show tangible results that indicate to the customer that your solution is the right one.

This step could also include customer interviews. Make sure that the people being interviewed are key stakeholders in the purchase decision or deployment and use of the product or service that is being highlighted. Content writers should work off a set list of questions prepared in advance. It can be helpful to share these with the interviewees beforehand so they have time to consider and craft their responses. One of the best interview tactics to keep in mind is to ask questions where yes and no are not natural answers. This way, your subject will provide more open-ended responses that produce more meaningful content.

4. Choose the right format

There are a number of different ways to format a case study. Depending on what you hope to achieve, one style will be better than another. However, there are some common elements to include, such as:

• An engaging headline
• A subject and customer introduction
• The unique challenge or challenges the customer faced
• The solution the customer used to solve the problem
• The results achieved
• Data and statistics to back up claims of success
• A strong call to action (CTA) to engage with the vendor

It’s also important to note that while case studies are traditionally written as stories, they don’t have to be in a written format. Some companies choose to get more creative with their case studies and produce multimedia content, depending on their audience and objectives. Case study formats can include traditional print stories, interactive web or social content, data-heavy infographics, professionally shot videos, podcasts, and more.

5. Write your case study

We’ll go into more detail later about how exactly to write a case study, including templates and examples. Generally speaking, though, there are a few things to keep in mind when writing your case study.

• Be clear and concise. Readers want to get to the point of the story quickly and easily, and they’ll be looking to see themselves reflected in the story right from the start.
• Provide a big picture. Always make sure to explain who the client is, their goals, and how they achieved success in a short introduction to engage the reader.
• Construct a clear narrative. Stick to the story from the perspective of the customer and what they needed to solve instead of just listing product features or benefits.
• Leverage graphics. Incorporating infographics, charts, and sidebars can be a more engaging and eye-catching way to share key statistics and data in readable ways.
• Offer the right amount of detail. Most case studies are one or two pages with clear sections that a reader can skim to find the information most important to them.
• Include data to support claims. Show real results — both facts and figures and customer quotes — to demonstrate credibility and prove the solution works.

6. Promote your story

Marketers have a number of options for distribution of a freshly minted case study. Many brands choose to publish case studies on their website and post them on social media. This can help support SEO and organic content strategies while also boosting company credibility and trust as visitors see that other businesses have used the product or service.

Marketers are always looking for quality content they can use for lead generation. Consider offering a case study as gated content behind a form on a landing page or as an offer in an email message. One great way to do this is to summarize the content and tease the full story available for download after the user takes an action.

Sales teams can also leverage case studies, so be sure they are aware that the assets exist once they’re published. Especially when it comes to larger B2B sales, companies often ask for examples of similar customer challenges that have been solved.

Now that you’ve learned a bit about case studies and what they should include, you may be wondering how to start creating great customer story content. Here are a couple of templates you can use to structure your case study.

Template 1 — Challenge-solution-result format

• Start with an engaging title. This should be fewer than 70 characters long for SEO best practices. One of the best ways to approach the title is to include the customer’s name and a hint at the challenge they overcame in the end.
• Create an introduction. Lead with an explanation as to who the customer is, the need they had, and the opportunity they found with a specific product or solution. Writers can also suggest the success the customer experienced with the solution they chose.
• Present the challenge. This should be several paragraphs long and explain the problem the customer faced and the issues they were trying to solve. Details should tie into the company’s products and services naturally. This section needs to be the most relatable to the reader so they can picture themselves in a similar situation.
• Share the solution. Explain which product or service offered was the ideal fit for the customer and why. Feel free to delve into their experience setting up, purchasing, and onboarding the solution.
• Explain the results. Demonstrate the impact of the solution they chose by backing up their positive experience with data. Fill in with customer quotes and tangible, measurable results that show the effect of their choice.
• Ask for action. Include a CTA at the end of the case study that invites readers to reach out for more information, try a demo, or learn more — to nurture them further in the marketing pipeline. What you ask of the reader should tie directly into the goals that were established for the case study in the first place.

Template 2 — Data-driven format

• Start with an engaging title. Be sure to include a statistic or data point in the first 70 characters. Again, it’s best to include the customer’s name as part of the title.
• Create an overview. Share the customer’s background and a short version of the challenge they faced. Present the reason a particular product or service was chosen, and feel free to include quotes from the customer about their selection process.
• Present data point 1. Isolate the first metric that the customer used to define success and explain how the product or solution helped to achieve this goal. Provide data points and quotes to substantiate the claim that success was achieved.
• Present data point 2. Isolate the second metric that the customer used to define success and explain what the product or solution did to achieve this goal. Provide data points and quotes to substantiate the claim that success was achieved.
• Present data point 3. Isolate the final metric that the customer used to define success and explain what the product or solution did to achieve this goal. Provide data points and quotes to substantiate the claim that success was achieved.
• Summarize the results. Reiterate the fact that the customer was able to achieve success thanks to a specific product or service. Include quotes and statements that reflect customer satisfaction and suggest they plan to continue using the solution.
• Ask for action. Include a CTA at the end of the case study that asks readers to reach out for more information, try a demo, or learn more — to further nurture them in the marketing pipeline. Again, remember that this is where marketers can look to convert their content into action with the customer.

While templates are helpful, seeing a case study in action can also be a great way to learn. Here are some examples of how Adobe customers have experienced success.

Juniper Networks

One example is the Adobe and Juniper Networks case study , which puts the reader in the customer’s shoes. The beginning of the story quickly orients the reader so that they know exactly who the article is about and what they were trying to achieve. Solutions are outlined in a way that shows Adobe Experience Manager is the best choice and a natural fit for the customer. Along the way, quotes from the client are incorporated to help add validity to the statements. The results in the case study are conveyed with clear evidence of scale and volume using tangible data.

The story of Lenovo’s journey with Adobe is one that spans years of planning, implementation, and rollout. The Lenovo case study does a great job of consolidating all of this into a relatable journey that other enterprise organizations can see themselves taking, despite the project size. This case study also features descriptive headers and compelling visual elements that engage the reader and strengthen the content.

Tata Consulting

When it comes to using data to show customer results, this case study does an excellent job of conveying details and numbers in an easy-to-digest manner. Bullet points at the start break up the content while also helping the reader understand exactly what the case study will be about. Tata Consulting used Adobe to deliver elevated, engaging content experiences for a large telecommunications client of its own — an objective that’s relatable for a lot of companies.

Case studies are a vital tool for any marketing team as they enable you to demonstrate the value of your company’s products and services to others. They help marketers do their job and add credibility to a brand trying to promote its solutions by using the experiences and stories of real customers.

When you’re ready to get started with a case study:

• Think about a few goals you’d like to accomplish with your content.
• Make a list of successful clients that would be strong candidates for a case study.
• Reach out to the client to get their approval and conduct an interview.
• Gather the data to present an engaging and effective customer story.

Adobe can help

There are several Adobe products that can help you craft compelling case studies. Adobe Experience Platform helps you collect data and deliver great customer experiences across every channel. Once you’ve created your case studies, Experience Platform will help you deliver the right information to the right customer at the right time for maximum impact.

To learn more, watch the Adobe Experience Platform story .

Keep in mind that the best case studies are backed by data. That’s where Adobe Real-Time Customer Data Platform and Adobe Analytics come into play. With Real-Time CDP, you can gather the data you need to build a great case study and target specific customers to deliver the content to the right audience at the perfect moment.

Watch the Real-Time CDP overview video to learn more.

Finally, Adobe Analytics turns real-time data into real-time insights. It helps your business collect and synthesize data from multiple platforms to make more informed decisions and create the best case study possible.

Request a demo to learn more about Adobe Analytics.

https://business.adobe.com/blog/perspectives/b2b-ecommerce-10-case-studies-inspire-you

https://business.adobe.com/blog/basics/business-case

https://business.adobe.com/blog/basics/what-is-real-time-analytics

3 Successful ERP Implementation Case Studies

What do a maker of grooming products for men, a fast-growing moving business and a specialized shipping logistics provider have in common? All three had intractable challenges that were addressed by an enterprise resource planning (ERP) system.

In all cases, their journeys started with ERP implementations. Once up and running, these disparate companies got the results they needed: more sales without adding new employees, faster and more accurate payroll processing and the ability to deliver tens of thousands of orders per day without delays or errors, respectively.

What Is an ERP Implementation?

An ERP implementation is the process of installing enterprise resource planning (ERP) software, and then making sure your team is making the most of the technology. The process includes three main phases:

• Installing onsite software and/or contracting with a SaaS provider. Often, companies seek help from the vendor or an implementation partner specializing in ERP, the vertical industry or specialized use cases.
• Migrating data from your previous system or systems into the new ERP.
• Training the finance team to use the new software. That process may include knowledge transfer from employees who previously used the system or formal education from your implementation partner.

The success rate ERP implementations is improving thanks to a focus on user experience, specifically simpler interfaces, more configuration flexibility, mobile functionality and increased automation. In fact, IDC’s 2021 SaaSPath Survey showed that "ease of use" is the most important vendor selection criteria for those purchasing finance applications.

Best practices to ensure success include organizing a strong implementation team of leaders from all areas of your company, from sales to accounting to operations. Ask them: What problems would we like this new system to resolve?

When choosing the ERP, it’s important to ensure it can easily integrate with other financial systems, and those of your key partners, and that you can be up and running in a reasonable timeframe, preferably 90 days or less. Finally, make sure your finance policies and procedures are aligned to make the most of the new system.

Key Takeaways

An ERP implementation is a process: ERP vendors and reviewers alike recommend carefully evaluating ERP vendors and/or partners before beginning, as well as taking things one step at a time, setting expectations with realistic planning timeframes and a comprehensive checklist mapped to your company’s goals for the system.

Let’s look at three companies that found success with their ERP implementation projects, and their key lessons learned.

ERP Case Study #1: Fulton & Roark

Fulton & Roark , a retailer of men’s grooming products, is an example of a successful ERP implementation.

Prior to upgrading to full-featured ERP, the North Carolina-based business tracked its inventory in a spreadsheet and its financial data in desktop accounting software, Sage Live. When the company began doubling sales year-over-year, leadership felt its current processes weren’t keeping up. Spreadsheets couldn’t account for changing inventory costs, and the accounting software didn’t have the workflows necessary to record the cost of goods sold (COGS) , an important financial metric.

As a result, the Fulton & Roark team did double data entry — manually.

To centralize all work in one place, the company’s co-founders implemented NetSuite ERP . After a three-week implementation process, changes were immediate, according to team members. Finally, the Fulton & Roark team was able to:

• Catch and correct bookkeeping mistakes related to inventory.
• Stop working with external accountants, growing both unit and dollar volumes significantly with no extra headcount.
• Increase sales roughly 50% year-over-year without increasing headcount.
• Get a more accurate picture of margins and inventory, which helped grow its ecommerce operation.
• ERP implementations don’t have to drag on — Fulton & Roark’s team was up and working in about 20 days.
• The company’s story also emphasizes a major success factor: Getting management committed to an ERP project. In this case, the co-founders initiated the project, which consultants say often spurs employee adoption.

ERP Case Study #2: N&N Moving Supplies

N&N Moving Supplies, a family-run distributor of moving equipment and supplies, successfully implemented an ERP system after expanding from one location in Georgia to three locations in multiple states and more than quadrupling its workforce.

Running on QuickBooks and a third-party payroll provider, it was nearly impossible to keep accurate time records and reconcile payroll with general ledger accounts. So, N&N turned to ERP to manage its accounting and payroll processes. During the implementation process for NetSuite ERP, N&N worked with a NetSuite partner, NOVAtime, to add a time-clock solution to the main system.

With this in place, N&N was able to:

• Reduce payroll processing time by 84%.
• Balance accounts faster.
• Improve the accuracy of hours and vacation time counts.
• See labor-cost trends across its three locations.
• Give employees access to personalized ERP dashboards on iPads at each of its sites.
• N&N’s case study nods to another major implementation success factor: managing employee morale. ERP projects often fail when execs and other employees don’t feel a sense of buy-in. By providing personalized dashboards — and ideally explaining their benefits to employees before the implementation — N&N avoided this pitfall.
• The case study also shows that a third-party partner can be helpful in tailoring the ERP system to precisely fit your company’s needs.

ERP Case Study #3: Green Rabbit

Once upon a time, two friends launched a candy wholesale business. Then, they ran into a problem: shipping melt-prone chocolate bars during summer months. After the pals designed a heat-sensitive supply chain, other food companies turned to them for help. The duo rebranded as Green Rabbit, a supply chain logistics provider specializing in fast delivery of perishable goods.

That shift in strategy tested the company’s current processes, which relied on QuickBooks, Excel and email. Green Rabbit’s multiple databases couldn’t communicate with one another, making real-time data analysis impossible. The warehouse was often waiting on help from the IT team, too.

Green Rabbit chose NetSuite ERP, and the NetSuite professional services team got the company up and running on the system in three months.

Green Rabbit now:

• Helps customers ship candy, snacks and more across the country in 24 hours from one of three different warehouses, without inventory errors.
• Delivers tens of thousands of orders per day, without delays.
• Gets guaranteed error-free data from its ERP system, instead of risking errors from manually entered data.
• Could triple its order volume, if desired, without impact to its systems.
• When implemented correctly, ERP makes it easier to manage a complicated supply chain, as Green Rabbit’s case study shows.
• ERP is pivotal to growth. Green Rabbit implemented ERP after it grew substantially and because it anticipated further expansion.

Many growing companies like those profiled here ask: “What is the business case for ERP?”

Besides the takeaways from our case studies, it boils down to agility. When finance teams stop peering at dozens of spreadsheets and paper records, they can get strategic. By helping executive and business colleagues visualize data, growing firms can forecast future trends and adapt quickly to beat competitors.

ERP Success Factors

ERP systems can revolutionize businesses and ease their paths to growth, but successful applications of them share some common traits. The first is employee adoption and morale, which requires encouraging buy-in by presenting the ERP in a way in which it will directly benefit the end users. Providing dashboards and solutions tailored to each business makes this easier to do — and tagging in a third-party partner to help with a tailored implementation is an effective way to accomplish this. Also important is planning and executing the implementation efficiently , while having realistic expectations for the timeline.

#1 Cloud ERP Software

What Are the Challenges of ERP?

Implementing an ERP can mean overcoming obstacles. Two of the most commonly cited ERP challenges are:

Choosing the right vendor. Obviously, choosing an ERP system that doesn’t fit your company’s needs will likely result in a failed implementation. Many advisers recommend evaluating around five vendors before choosing which ERP to purchase.

Companies should choose a vendor that has experience with your company’s vertical, type (product or service) and size; can provide a hands-on demo; understands the tax laws and regulations that apply to your business; and is sure to be in business in five years.

• Getting employees to support the change. As mentioned, teams may resist the transition to ERP if you don’t show them why the new system will be helpful to them. Explain how ERP will make each team’s everyday tasks easier before the implementation begins and ensure plenty of time for training during the process.

Why Do ERP Implementations Fail?

ERP implementations can fail without proper risk management. In other words, leaders need to anticipate what might derail the project, then plan.

In one often-cited case, for example, a multinational distributor’s implementation epically failed, resulting in major shipping delays and lost sales. It turns out the company had operational issues before the implementation even began, as it had just acquired another company and was having trouble integrating the new subsidiary’s operations into its own. Company leadership should have managed that risk by identifying and fixing those operational issues before the ERP implementation began.

ERP implementation is a process. It requires lots of planning to ensure success. However, your company isn’t the first to go through it. Consider case studies of successful implementations, choose the right partner and your business will enjoy the benefits of the system, quickly.

If you’re considering an ERP implementation, schedule a consultation with NetSuite .

ERP Implementation Plan: Methodology and Process

Enterprise resources planning (ERP) solutions are designed to support an organization in just about every aspect of business operations by centralizing data and through automation. These platforms control internal…

More On This

Trending Articles

Learn How NetSuite Can Streamline Your Business

NetSuite has packaged the experience gained from tens of thousands of worldwide deployments over two decades into a set of leading practices that pave a clear path to success and are proven to deliver rapid business value. With NetSuite, you go live in a predictable timeframe — smart, stepped implementations begin with sales and span the entire customer lifecycle, so there’s continuity from sales to services to support.

Before you go...

Discover the products that 37,000+ customers depend on to fuel their growth.

Before you go. Talk with our team or check out these resources.

Want to set up a chat later? Let us do the lifting.

NetSuite ERP

Explore what NetSuite ERP can do for you.

Business Guide

Complete Guide to Cloud ERP Implementation

• Skip to content
• Accessibility Policy
• QUICK LINKS
• Oracle Cloud Infrastructure
• Oracle Fusion Cloud Applications
• Oracle Database
• Download Java
• Careers at Oracle

• Create an Account

3 ERP Implementation Case Studies

Mark Jackley | Content Strategist | January 10, 2023

In This Article

Why Do ERP Implementations Fail?

Keys to successful erp implementation projects, your erp implementation starts by choosing the right system, erp implementation case study faqs.

Enterprise resource planning (ERP) software connects the most important functions of businesses—including financials and asset management, supply chain management, customer relationship management (CRM), and human resources. That’s why an ERP implementation is such a big deal. The payoff can be substantial: A single system to connect and standardize data in real-time gives teams across the business a shared view of performance—and one source of truth to improve decision-making and planning.

However, ERP implementations have potential pitfalls. If your project team fails to create a vision to guide the project, instill a spirit of teamwork among everyone involved, and communicate its objectives to employees, you might not solve your business challenges and get the results you need.

The following three case studies spotlight companies that did things right. By borrowing from their playbooks, your business can also enjoy the benefits of a successful ERP implementation.

1. A clear vision: Discover Financial Services moves to cloud ERP

In 2019, Discover Financial Services, the third-largest credit card brand in the United States, was running seven highly customized on-premises ERP systems. Their IT landscape was complex and lacked standardization, which resulted in slow and sometimes inaccurate reporting. Moreover, the accounting team and their counterparts in financial analysis and planning needed more scalable systems to support their growing workloads. After evaluating several vendors, Discover chose to simplify its environment by consolidating systems on Oracle Fusion Cloud ERP, integrated with Oracle Fusion Cloud Enterprise Performance Management (EPM).

The project team had a clear vision: The ERP implementation was more than an IT upgrade. It was an opportunity to create and sustain business value companywide, generating “more data and higher quality,” says Michelle Green, vice president, business technology: strategy, transformation, and governance. “Ultimately, the functionality and automation help us make more effective enterprisewide decisions.”

This vision helped Discover complete the ERP implementation on time and on budget, despite working remotely during the COVID-19 pandemic. “There are two things I look for in an implementation,” says Green. “The first is when people say the implementation was a nonevent. The second is when people start giving unsolicited feedback that they’re loving the systems. On day 15, post go-live, we heard both.”

The modernized ERP platform has paid off. With faster access to financial information, Discover employees companywide can more effectively manage expenses . Lines of business enjoy more accurate planning projections and cost analysis. With standardized functionality replacing customization, employees spend less time deciphering technology and more time achieving results. And with quarterly cloud functionality updates delivered automatically, Discover can continue to enhance its processes and ways of working.

2. Teamwork and trust: ERP helps Hormel Foods unify the business

With a long history of making strategic acquisitions, Hormel Foods owns more than 50 iconic brands, such as Dinty Moore, Planters Peanuts, SPAM, and Skippy Peanut Butter. But at one time, it also owned a patchwork of different IT systems.

“We had various systems for various companies that did not interface well together,” says Jim Sheehan, executive vice president and chief financial officer, Hormel Foods. “They did not provide us with a clear view of the company's performance, were difficult to maintain, and honestly, had become a burden to our competitiveness.”

In some cases, the human capital management (HCM) and ERP applications hadn’t been upgraded in more than 15 years. Some systems depended on a single person—“whoever remained who understood the code it was written on,” Sheehan says. Adding to the complexity, Hormel Foods had customized many of its applications to meet the needs of individual brands.

To simplify business processes and improve data quality, Hormel worked with KPMG to implement Oracle Cloud ERP, part of the Oracle Cloud Applications suite that now also supports Hormel’s supply chain management, enterprise performance management, and human resources.

“We had to rely on teamwork,” says Assistant Controller Eldon Quam. “We had Oracle, KPMG, and Hormel Foods people all working together. If you weren’t familiar with the group, you wouldn’t have known the difference between a Hormel Foods person and someone who worked for another company.”

When the ERP system went live, Hormel gained a unified system to standardize business processes, data, and decision-making. “We have visibility into every business from the same point of view,” Sheehan says.

Hormel Foods also has a forecasting model to pinpoint growth opportunities across its many brands. “In the past, we looked at things brand by brand,” says Jana Haynes, vice president and controller. “That’s still important to do, but now that we’re on a centralized platform, we’re able to notice things that weren’t apparent before.”

With procurement in all countries except China and Brazil now on the same platform, it’s easier to analyze vendors as well. For instance, the company discovered that some of its brands paid more than others for the same item. Hormel can also pinpoint the number of vendors that are the sole source of items its brands rely on. Knowing this allows the company to explore alternative sources and be ready should a vendor prove unable to meet demand.

With the entire company running on one ERP system, every brand does financial reporting in the same way. “Whether it’s their cash flow, their income statement, or their balance sheet items,” Sheehan says. In other words, all that teamwork enhanced their ability to operate as one team.

3. Employee buy-in: Taylor Corporation rallies support for digital transformation

A diversified global company that operates printing, data, and marketing businesses, Taylor Corporation embarked on a four-to-five-year effort to implement an ERP solution and modernize its back office, moving financial, supply chain, and manufacturing applications to Oracle Cloud. When all phases of the ERP implementation are completed, Taylor estimates it will see between $20 to $30 million in total cost savings.

CEO Charlie Whitaker has made it his personal mission to see the program succeed. On a kickoff video conference early in the implementation, Whitaker rallied the company’s 10,000 employees, urging them to see the program as key to Taylor’s growth and prosperity. He also led steering-committee meetings and working sessions with project team members.

“If you talk with Charlie, it's not an option to participate. It’s a requirement: Don’t debate it, participate in it, which I think is a fantastic message,” says Jenn Warpinski, Taylor’s vice president of enterprise transformation.

Several Taylor managers who were initially skeptical now are leading the way in bringing their colleagues around. “They’re now jumping in, and they’re doing the talking, they’re doing the change management,” Warpinski says.

Workforce support is necessary to succeed with such a transformational project. Indeed, the scope was significant: Taylor needed to consolidate data and business processes across 85 different systems, some of them no longer supported by vendors. Taylor wanted to make timely and informed financial and supplier decisions, automate manual processes that were costly and time consuming, and make it easier for customers to operate across the company’s business units.

For example, one of Taylor’s five multisubsidiary enterprise groups had customer information sitting in 16 different systems. That made it difficult for the group to cross-sell products across business lines. (Taylor units sell everything from printed wedding invitations to digital marketing services.)

Before phase one of the implementation—a global rollout of cloud financials completed in January 2022—it would take Taylor’s finance teams a couple of weeks to assemble a consolidated view of outstanding accounts receivable across the company. That data is now available for executives to act on in real time.

Now that Taylor is also live on cloud procurement applications, it can negotiate better volume discounts with approved indirect suppliers, while end users can get approvals and execute orders for supplies with just a couple of clicks.

One leading cause is failing to practice contingency planning. At the outset, implementation teams should identify likely causes of delays or cost overruns and plan accordingly. For instance, are there existing vulnerabilities in your company’s business model that need to be fixed before launching the project?

Unrealistic expectations are another reason for failure. If your timetable shrinks but your list of deliverables doesn’t, the ERP implementation could be doomed from the start, or at the very least become chaotic and expensive.

Some careful whiteboarding can help you avoid these potential problems. You can enjoy a successful ERP implementation with the right ERP provider, executive support, and a willingness to learn from other businesses’ successes and mistakes and a willingness to learn from other businesses’ successes and mistakes, you can enjoy a successful ERP implementation.

Besides crafting a vision, working as one team, and enlisting the support and patience of employees at all levels, other factors can lead to a successful ERP implementation. First, you’ll need to clearly define the project’s scope, objectives, and budget. A project team with the right expertise is another must. So are having plans for migrating data to the new system as well as a realistic timeframe for going live. A detailed change management plan that includes user training and education, such as the one Taylor created, is perhaps most critical to success.

Another ERP success factor: The system you implement should be easy for people to use. After all, simplifying the IT landscape and the user experience is a critical objective of any ERP system. If your implementation unfolds in stages, you can conduct periodic surveys to see if employees think the system is user friendly.

Your Complete Guide to Modern ERP

Discover what defines a modern ERP solution and how cloud ERP drives business agility and innovation.

Even if you devise a flawless implementation plan, it won’t yield great results if you choose the wrong ERP system. Look for a solution that can not only integrate disparate data and business processes but can also integrate with existing systems and applications your business will continue to run. Additionally, you’ll want capabilities that are purpose-built for functions, such as cash management or procurement, giving you faster and more accurate information to improve decision-making. It’s also smart to choose a system that can both monetize and measure business value as you plan new products and set pricing strategies.

Of course, your project management team should thoroughly assess ERP vendors. It’s important to listen carefully to what they can actually deliver, looking beyond basic features and capabilities. For example, what is their support offering like? What kind of training do they provide? What does the future look like—is the vendor truly invested in the emerging technologies you’ll need to compete? You’ll also want to hear what industry analysts, such as Gartner, say about cloud ERP solutions, including Oracle Fusion Cloud ERP.

Lastly, consider what else an ERP system can offer your company. Oracle’s solution, for example, includes built-in risk management tools that use AI and machine learning to strengthen financial controls. It also can integrate with your customer relationship management (CRM) system, so your front- and back-office teams can seamlessly share data.

What are the top challenges of ERP implementations?

Three of the most common challenges are:

• Defining a vision of what the ERP system needs to do and how to manage the implementation—who should lead it and set priorities
• Engendering trust and teamwork between the project team, your ERP provider, and your consulting partner
• Communicating with employees about how the implementation will benefit them, freeing up time from mundane tasks to do more interesting work

See how Oracle Cloud ERP gives you the agility to adopt new business models and processes quickly, helping you reduce costs, sharpen forecasts, and innovate faster.

• - Google Chrome

Intended for healthcare professionals

• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Implementation...

Implementation research: what it is and how to do it

• Related content
• Peer review
• David H Peters , professor 1 ,
• Taghreed Adam , scientist 2 ,
• Olakunle Alonge , assistant scientist 1 ,
• Irene Akua Agyepong , specialist public health 3 ,
• Nhan Tran , manager 4
• 1 Johns Hopkins University Bloomberg School of Public Health, Department of International Health, 615 N Wolfe St, Baltimore, MD 21205, USA
• 2 Alliance for Health Policy and Systems Research, World Health Organization, CH-1211 Geneva 27, Switzerland
• 3 University of Ghana School of Public Health/Ghana Health Service, Accra, Ghana
• 4 Alliance for Health Policy and Systems Research, Implementation Research Platform, World Health Organization, CH-1211 Geneva 27, Switzerland
• Correspondence to: D H Peters  dpeters{at}jhsph.edu
• Accepted 8 October 2013

Implementation research is a growing but not well understood field of health research that can contribute to more effective public health and clinical policies and programmes. This article provides a broad definition of implementation research and outlines key principles for how to do it

The field of implementation research is growing, but it is not well understood despite the need for better research to inform decisions about health policies, programmes, and practices. This article focuses on the context and factors affecting implementation, the key audiences for the research, implementation outcome variables that describe various aspects of how implementation occurs, and the study of implementation strategies that support the delivery of health services, programmes, and policies. We provide a framework for using the research question as the basis for selecting among the wide range of qualitative, quantitative, and mixed methods that can be applied in implementation research, along with brief descriptions of methods specifically suitable for implementation research. Expanding the use of well designed implementation research should contribute to more effective public health and clinical policies and programmes.

Defining implementation research

Implementation research attempts to solve a wide range of implementation problems; it has its origins in several disciplines and research traditions (supplementary table A). Although progress has been made in conceptualising implementation research over the past decade, 1 considerable confusion persists about its terminology and scope. 2 3 4 The word “implement” comes from the Latin “implere,” meaning to fulfil or to carry into effect. 5 This provides a basis for a broad definition of implementation research that can be used across research traditions and has meaning for practitioners, policy makers, and the interested public: “Implementation research is the scientific inquiry into questions concerning implementation—the act of carrying an intention into effect, which in health research can be policies, programmes, or individual practices (collectively called interventions).”

Implementation research can consider any aspect of implementation, including the factors affecting implementation, the processes of implementation, and the results of implementation, including how to introduce potential solutions into a health system or how to promote their large scale use and sustainability. The intent is to understand what, why, and how interventions work in “real world” settings and to test approaches to improve them.

Principles of implementation research

Implementation research seeks to understand and work within real world conditions, rather than trying to control for these conditions or to remove their influence as causal effects. This implies working with populations that will be affected by an intervention, rather than selecting beneficiaries who may not represent the target population of an intervention (such as studying healthy volunteers or excluding patients who have comorbidities).

Context plays a central role in implementation research. Context can include the social, cultural, economic, political, legal, and physical environment, as well as the institutional setting, comprising various stakeholders and their interactions, and the demographic and epidemiological conditions. The structure of the health systems (for example, the roles played by governments, non-governmental organisations, other private providers, and citizens) is particularly important for implementation research on health.

Implementation research is especially concerned with the users of the research and not purely the production of knowledge. These users may include managers and teams using quality improvement strategies, executive decision makers seeking advice for specific decisions, policy makers who need to be informed about particular programmes, practitioners who need to be convinced to use interventions that are based on evidence, people who are influenced to change their behaviour to have a healthier life, or communities who are conducting the research and taking action through the research to improve their conditions (supplementary table A). One important implication is that often these actors should be intimately involved in the identification, design, and conduct phases of research and not just be targets for dissemination of study results.

Implementation outcome variables

Implementation outcome variables describe the intentional actions to deliver services. 6 These implementation outcome variables—acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, coverage, and sustainability—can all serve as indicators of the success of implementation (table 1 ⇓ ). Implementation research uses these variables to assess how well implementation has occurred or to provide insights about how this contributes to one’s health status or other important health outcomes.

 Implementation outcome variables

• View inline

Implementation strategies

Curran and colleagues defined an “implementation intervention” as a method to “enhance the adoption of a ‘clinical’ intervention,” such as the use of job aids, provider education, or audit procedures. 7 The concept can be broadened to any type of strategy that is designed to support a clinical or population and public health intervention (for example, outreach clinics and supervision checklists are implementation strategies used to improve the coverage and quality of immunisation).

A review of ways to improve health service delivery in low and middle income countries identified a wide range of successful implementation strategies (supplementary table B). 8 Even in the most resource constrained environments, measuring change, informing stakeholders, and using information to guide decision making were found to be critical to successful implementation.

Implementation influencing variables

Other factors that influence implementation may need to be considered in implementation research. Sabatier summarised a set of such factors that influence policy implementation (clarity of objectives, causal theory, implementing personnel, support of interest groups, and managerial authority and resources). 9

The large array of contextual factors that influence implementation, interact with each other, and change over time highlights the fact that implementation often occurs as part of complex adaptive systems. 10 Some implementation strategies are particularly suitable for working in complex systems. These include strategies to provide feedback to key stakeholders and to encourage learning and adaptation by implementing agencies and beneficiary groups. Such strategies have implications for research, as the study methods need to be sufficiently flexible to account for changes or adaptations in what is actually being implemented. 8 11 Research designs that depend on having a single and fixed intervention, such as a typical randomised controlled trial, would not be an appropriate design to study phenomena that change, especially when they change in unpredictable and variable ways.

Another implication of studying complex systems is that the research may need to use multiple methods and different sources of information to understand an implementation problem. Because implementation activities and effects are not usually static or linear processes, research designs often need to be able to observe and analyse these sometimes iterative and changing elements at several points in time and to consider unintended consequences.

Implementation research questions

As in other types of health systems research, the research question is the king in implementation research. Implementation research takes a pragmatic approach, placing the research question (or implementation problem) as the starting point to inquiry; this then dictates the research methods and assumptions to be used. Implementation research questions can cover a wide variety of topics and are frequently organised around theories of change or the type of research objective (examples are in supplementary table C). 12 13

Implementation research can overlap with other types of research used in medicine and public health, and the distinctions are not always clear cut. A range of implementation research exists, based on the centrality of implementation in the research question, the degree to which the research takes place in a real world setting with routine populations, and the role of implementation strategies and implementation variables in the research (figure ⇓ ).

Spectrum of implementation research 33

• Download figure
• Open in new tab
• Download powerpoint

A more detailed description of the research question can help researchers and practitioners to determine the type of research methods that should be used. In table 2 ⇓ , we break down the research question first by its objective: to explore, describe, influence, explain, or predict. This is followed by a typical implementation research question based on each objective. Finally, we describe a set of research methods for each type of research question.

 Type of implementation research objective, implementation question, and research methods

Much of evidence based medicine is concerned with the objective of influence, or whether an intervention produces an expected outcome, which can be broken down further by the level of certainty in the conclusions drawn from the study. The nature of the inquiry (for example, the amount of risk and considerations of ethics, costs, and timeliness), and the interests of different audiences, should determine the level of uncertainty. 8 14 Research questions concerning programmatic decisions about the process of an implementation strategy may justify a lower level of certainty for the manager and policy maker, using research methods that would support an adequacy or plausibility inference. 14 Where a high risk of harm exists and sufficient time and resources are available, a probability study design might be more appropriate, in which the result in an area where the intervention is implemented is compared with areas without implementation with a low probability of error (for example, P< 0.05). These differences in the level of confidence affect the study design in terms of sample size and the need for concurrent or randomised comparison groups. 8 14

Implementation specific research methods

A wide range of qualitative and quantitative research methods can be used in implementation research (table 2 ⇑ ). The box gives a set of basic questions to guide the design or reporting of implementation research that can be used across methods. More in-depth criteria have also been proposed to assess the external validity or generalisability of findings. 15 Some research methods have been developed specifically to deal with implementation research questions or are particularly suitable to implementation research, as identified below.

Key questions to assess research designs or reports on implementation research 33

Does the research clearly aim to answer a question concerning implementation?

Does the research clearly identify the primary audiences for the research and how they would use the research?

Is there a clear description of what is being implemented (for example, details of the practice, programme, or policy)?

Does the research involve an implementation strategy? If so, is it described and examined in its fullness?

Is the research conducted in a “real world” setting? If so, is the context and sample population described in sufficient detail?

Does the research appropriately consider implementation outcome variables?

Does the research appropriately consider context and other factors that influence implementation?

Does the research appropriately consider changes over time and the level of complexity of the system, including unintended consequences?

Pragmatic trials

Pragmatic trials, or practical trials, are randomised controlled trials in which the main research question focuses on effectiveness of an intervention in a normal practice setting with the full range of study participants. 16 This may include pragmatic trials on new healthcare delivery strategies, such as integrated chronic care clinics or nurse run community clinics. This contrasts with typical randomised controlled trials that look at the efficacy of an intervention in an “ideal” or controlled setting and with highly selected patients and standardised clinical outcomes, usually of a short term nature.

Effectiveness-implementation hybrid trials

Effectiveness-implementation hybrid designs are intended to assess the effectiveness of both an intervention and an implementation strategy. 7 These studies include components of an effectiveness design (for example, randomised allocation to intervention and comparison arms) but add the testing of an implementation strategy, which may also be randomised. This might include testing the effectiveness of a package of delivery and postnatal care in under-served areas, as well testing several strategies for providing the care. Whereas pragmatic trials try to fix the intervention under study, effectiveness-implementation hybrids also intervene and/or observe the implementation process as it actually occurs. This can be done by assessing implementation outcome variables.

Quality improvement studies

Quality improvement studies typically involve a set of structured and cyclical processes, often called the plan-do-study-act cycle, and apply scientific methods on a continuous basis to formulate a plan, implement the plan, and analyse and interpret the results, followed by an iteration of what to do next. 17 18 The focus might be on a clinical process, such as how to reduce hospital acquired infections in the intensive care unit, or management processes such as how to reduce waiting times in the emergency room. Guidelines exist on how to design and report such research—the Standards for Quality Improvement Reporting Excellence (SQUIRE). 17

Speroff and O’Connor describe a range of plan-do-study-act research designs, noting that they have in common the assessment of responses measured repeatedly and regularly over time, either in a single case or with comparison groups. 18 Balanced scorecards integrate performance measures across a range of domains and feed into regular decision making. 19 20 Standardised guidance for using good quality health information systems and health facility surveys has been developed and often provides the sources of information for these quasi-experimental designs. 21 22 23

Participatory action research

Participatory action research refers to a range of research methods that emphasise participation and action (that is, implementation), using methods that involve iterative processes of reflection and action, “carried out with and by local people rather than on them.” 24 In participatory action research, a distinguishing feature is that the power and control over the process rests with the participants themselves. Although most participatory action methods involve qualitative methods, quantitative and mixed methods techniques are increasingly being used, such as for participatory rural appraisal or participatory statistics. 25 26

Mixed methods

Mixed methods research uses both qualitative and quantitative methods of data collection and analysis in the same study. Although not designed specifically for implementation research, mixed methods are particularly suitable because they provide a practical way to understand multiple perspectives, different types of causal pathways, and multiple types of outcomes—all common features of implementation research problems.

Many different schemes exist for describing different types of mixed methods research, on the basis of the emphasis of the study, the sampling schemes for the different components, the timing and sequencing of the qualitative and quantitative methods, and the level of mixing between the qualitative and quantitative methods. 27 28 Broad guidance on the design and conduct of mixed methods designs is available. 29 30 31 A scheme for good reporting of mixed methods studies involves describing the justification for using a mixed methods approach to the research question; describing the design in terms of the purpose, priority, and sequence of methods; describing each method in terms of sampling, data collection, and analysis; describing where the integration has occurred, how it has occurred, and who has participated in it; describing any limitation of one method associated with the presence of the other method; and describing any insights gained from mixing or integrating methods. 32

Implementation research aims to cover a wide set of research questions, implementation outcome variables, factors affecting implementation, and implementation strategies. This paper has identified a range of qualitative, quantitative, and mixed methods that can be used according to the specific research question, as well as several research designs that are particularly suited to implementation research. Further details of these concepts can be found in a new guide developed by the Alliance for Health Policy and Systems Research. 33

Summary points

Implementation research has its origins in many disciplines and is usefully defined as scientific inquiry into questions concerning implementation—the act of fulfilling or carrying out an intention

In health research, these intentions can be policies, programmes, or individual practices (collectively called interventions)

Implementation research seeks to understand and work in “real world” or usual practice settings, paying particular attention to the audience that will use the research, the context in which implementation occurs, and the factors that influence implementation

A wide variety of qualitative, quantitative, and mixed methods techniques can be used in implementation research, which are best selected on the basis of the research objective and specific questions related to what, why, and how interventions work

Implementation research may examine strategies that are specifically designed to improve the carrying out of health interventions or assess variables that are defined as implementation outcomes

Implementation outcomes include acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, coverage, and sustainability

Cite this as: BMJ 2013;347:f6753

Contributors: All authors contributed to the conception and design, analysis and interpretation, drafting the article, or revising it critically for important intellectual content, and all gave final approval of the version to be published. NT had the original idea for the article, which was discussed by the authors (except OA) as well as George Pariyo, Jim Sherry, and Dena Javadi at a meeting at the World Health Organization (WHO). DHP and OA did the literature reviews, and DHP wrote the original outline and the draft manuscript, tables, and boxes. OA prepared the original figure. All authors reviewed the draft article and made substantial revisions to the manuscript. DHP is the guarantor.

Funding: Funding was provided by the governments of Norway and Sweden and the UK Department for International Development (DFID) in support of the WHO Implementation Research Platform, which financed a meeting of authors and salary support for NT. DHP is supported by the Future Health Systems research programme consortium, funded by DFID for the benefit of developing countries (grant number H050474). The funders played no role in the design, conduct, or reporting of the research.

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support for the submitted work as described above; NT and TA are employees of the Alliance for Health Policy and Systems Research at WHO, which is supporting their salaries to work on implementation research; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Provenance and peer review: Invited by journal; commissioned by WHO; externally peer reviewed.

• ↵ Brownson RC, Colditz GA, Proctor EK, eds. Dissemination and implementation research in health: translating science to practice. Oxford University Press, 2012.
• ↵ Ciliska D, Robinson P, Armour T, Ellis P, Brouwers M, Gauld M, et al. Diffusion and dissemination of evidence-based dietary strategies for the prevention of cancer. Nutr J 2005 ; 4 (1): 13 . OpenUrl CrossRef PubMed
• ↵ Remme JHF, Adam T, Becerra-Posada F, D’Arcangues C, Devlin M, Gardner C, et al. Defining research to improve health systems. PLoS Med 2010 ; 7 : e1001000 . OpenUrl CrossRef PubMed
• ↵ McKibbon KA, Lokker C, Mathew D. Implementation research. 2012. http://whatiskt.wikispaces.com/Implementation+Research .
• ↵ The compact edition of the Oxford English dictionary. Oxford University Press, 1971.
• ↵ Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2010 ; 38 : 65 -76. OpenUrl
• ↵ Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 2012 ; 50 : 217 -26. OpenUrl CrossRef PubMed Web of Science
• ↵ Peters DH, El-Saharty S, Siadat B, Janovsky K, Vujicic M, eds. Improving health services in developing countries: from evidence to action. World Bank, 2009.
• ↵ Sabatier PA. Top-down and bottom-up approaches to implementation research. J Public Policy 1986 ; 6 (1): 21 -48. OpenUrl CrossRef
• ↵ Paina L, Peters DH. Understanding pathways for scaling up health services through the lens of complex adaptive systems. Health Policy Plan 2012 ; 27 : 365 -73. OpenUrl Abstract / FREE Full Text
• ↵ Gilson L, ed. Health policy and systems research: a methodology reader. World Health Organization, 2012.
• ↵ Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med 2012 ; 43 : 337 -50. OpenUrl CrossRef PubMed
• ↵ Improved Clinical Effectiveness through Behavioural Research Group (ICEBeRG). Designing theoretically-informed implementation interventions. Implement Sci 2006 ; 1 : 4 . OpenUrl CrossRef PubMed
• ↵ Habicht JP, Victora CG, Vaughn JP. Evaluation designs for adequacy, plausibility, and probability of public health programme performance and impact. Int J Epidemiol 1999 ; 28 : 10 -8. OpenUrl Abstract / FREE Full Text
• ↵ Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research. Eval Health Prof 2006 ; 29 : 126 -53. OpenUrl Abstract / FREE Full Text
• ↵ Swarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, et al, for the CONSORT and Pragmatic Trials in Healthcare (Practihc) Groups. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ 2008 ; 337 : a2390 . OpenUrl Abstract / FREE Full Text
• ↵ Davidoff F, Batalden P, Stevens D, Ogrince G, Mooney SE, for the SQUIRE Development Group. Publication guidelines for quality improvement in health care: evolution of the SQUIRE project. Qual Saf Health Care 2008 ; 17 (suppl I): i3 -9. OpenUrl Abstract / FREE Full Text
• ↵ Speroff T, O’Connor GT. Study designs for PDSA quality improvement research. Q Manage Health Care 2004 ; 13 (1): 17 -32. OpenUrl CrossRef
• ↵ Peters DH, Noor AA, Singh LP, Kakar FK, Hansen PM, Burnham G. A balanced scorecard for health services in Afghanistan. Bull World Health Organ 2007 ; 85 : 146 -51. OpenUrl CrossRef PubMed Web of Science
• ↵ Edward A, Kumar B, Kakar F, Salehi AS, Burnham G. Peters DH. Configuring balanced scorecards for measuring health systems performance: evidence from five years’ evaluation in Afghanistan. PLOS Med 2011 ; 7 : e1001066 . OpenUrl
• ↵ Health Facility Assessment Technical Working Group. Profiles of health facility assessment method, MEASURE Evaluation, USAID, 2008.
• ↵ Hotchkiss D, Diana M, Foreit K. How can routine health information systems improve health systems functioning in low-resource settings? Assessing the evidence base. MEASURE Evaluation, USAID, 2012.
• ↵ Lindelow M, Wagstaff A. Assessment of health facility performance: an introduction to data and measurement issues. In: Amin S, Das J, Goldstein M, eds. Are you being served? New tools for measuring service delivery. World Bank, 2008:19-66.
• ↵ Cornwall A, Jewkes R. “What is participatory research?” Soc Sci Med 1995 ; 41 : 1667 -76. OpenUrl CrossRef PubMed Web of Science
• ↵ Mergler D. Worker participation in occupational health research: theory and practice. Int J Health Serv 1987 ; 17 : 151 . OpenUrl Abstract / FREE Full Text
• ↵ Chambers R. Revolutions in development inquiry. Earthscan, 2008.
• ↵ Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. Sage Publications, 2011.
• ↵ Tashakkori A, Teddlie C. Mixed methodology: combining qualitative and quantitative approaches. Sage Publications, 2003.
• ↵ Leech NL, Onwuegbuzie AJ. Guidelines for conducting and reporting mixed research in the field of counseling and beyond. Journal of Counseling and Development 2010 ; 88 : 61 -9. OpenUrl CrossRef Web of Science
• ↵ Creswell JW. Mixed methods procedures. In: Research design: qualitative, quantitative and mixed methods approaches. 3rd ed. Sage Publications, 2009.
• ↵ Creswell JW, Klassen AC, Plano Clark VL, Clegg Smith K. Best practices for mixed methods research in the health sciences. National Institutes of Health, Office of Behavioral and Social Sciences Research, 2011.
• ↵ O’Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy 2008 ; 13 : 92 -8. OpenUrl Abstract / FREE Full Text
• ↵ Peters DH, Tran N, Adam T, Ghaffar A. Implementation research in health: a practical guide. Alliance for Health Policy and Systems Research, World Health Organization, 2013.
• Rogers EM. Diffusion of innovations. 5th ed. Free Press, 2003.
• Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci 2007 ; 2 : 40 . OpenUrl CrossRef PubMed
• Victora CG, Schellenberg JA, Huicho L, Amaral J, El Arifeen S, Pariyo G, et al. Context matters: interpreting impact findings in child survival evaluations. Health Policy Plan 2005 ; 20 (suppl 1): i18 -31. OpenUrl Abstract

• Case Study Evaluation Approach
• Learning Center

A case study evaluation approach can be an incredibly powerful tool for monitoring and evaluating complex programs and policies. By identifying common themes and patterns, this approach allows us to better understand the successes and challenges faced by the program. In this article, we’ll explore the benefits of using a case study evaluation approach in the monitoring and evaluation of projects, programs, and public policies.

Table of Contents

Introduction to Case Study Evaluation Approach

The advantages of a case study evaluation approach, types of case studies, potential challenges with a case study evaluation approach, guiding principles for successful implementation of a case study evaluation approach.

• Benefits of Incorporating the Case Study Evaluation Approach in the Monitoring and Evaluation of Projects and Programs

A case study evaluation approach is a great way to gain an in-depth understanding of a particular issue or situation. This type of approach allows the researcher to observe, analyze, and assess the effects of a particular situation on individuals or groups.

An individual, a location, or a project may serve as the focal point of a case study’s attention. Quantitative and qualitative data are frequently used in conjunction with one another.

It also allows the researcher to gain insights into how people react to external influences. By using a case study evaluation approach, researchers can gain insights into how certain factors such as policy change or a new technology have impacted individuals and communities. The data gathered through this approach can be used to formulate effective strategies for responding to changes and challenges. Ultimately, this monitoring and evaluation approach helps organizations make better decision about the implementation of their plans.

This approach can be used to assess the effectiveness of a policy, program, or initiative by considering specific elements such as implementation processes, outcomes, and impact. A case study evaluation approach can provide an in-depth understanding of the effectiveness of a program by closely examining the processes involved in its implementation. This includes understanding the context, stakeholders, and resources to gain insight into how well a program is functioning or has been executed. By evaluating these elements, it can help to identify areas for improvement and suggest potential solutions. The findings from this approach can then be used to inform decisions about policies, programs, and initiatives for improved outcomes.

It is also useful for determining if other policies, programs, or initiatives could be applied to similar situations in order to achieve similar results or improved outcomes. All in all, the case study monitoring evaluation approach is an effective method for determining the effectiveness of specific policies, programs, or initiatives. By researching and analyzing the successes of previous cases, this approach can be used to identify similar approaches that could be applied to similar situations in order to achieve similar results or improved outcomes.

A case study evaluation approach offers the advantage of providing in-depth insight into a particular program or policy. This can be accomplished by analyzing data and observations collected from a range of stakeholders such as program participants, service providers, and community members. The monitoring and evaluation approach is used to assess the impact of programs and inform the decision-making process to ensure successful implementation. The case study monitoring and evaluation approach can help identify any underlying issues that need to be addressed in order to improve program effectiveness. It also provides a reality check on how successful programs are actually working, allowing organizations to make adjustments as needed. Overall, a case study monitoring and evaluation approach helps to ensure that policies and programs are achieving their objectives while providing valuable insight into how they are performing overall.

By taking a qualitative approach to data collection and analysis, case study evaluations are able to capture nuances in the context of a particular program or policy that can be overlooked when relying solely on quantitative methods. Using this approach, insights can be gleaned from looking at the individual experiences and perspectives of actors involved, providing a more detailed understanding of the impact of the program or policy than is possible with other evaluation methodologies. As such, case study monitoring evaluation is an invaluable tool in assessing the effectiveness of a particular initiative, enabling more informed decision-making as well as more effective implementation of programs and policies.

Furthermore, this approach is an effective way to uncover experiential information that can help to inform the ongoing improvement of policy and programming over time All in all, the case study monitoring evaluation approach offers an effective way to uncover experiential information necessary to inform the ongoing improvement of policy and programming. By analyzing the data gathered from this systematic approach, stakeholders can gain deeper insight into how best to make meaningful and long-term changes in their respective organizations.

Case studies come in a variety of forms, each of which can be put to a unique set of evaluation tasks. Evaluators have come to a consensus on describing six distinct sorts of case studies, which are as follows: illustrative, exploratory, critical instance, program implementation, program effects, and cumulative.

Illustrative Case Study

An illustrative case study is a type of case study that is used to provide a detailed and descriptive account of a particular event, situation, or phenomenon. It is often used in research to provide a clear understanding of a complex issue, and to illustrate the practical application of theories or concepts.

An illustrative case study typically uses qualitative data, such as interviews, surveys, or observations, to provide a detailed account of the unit being studied. The case study may also include quantitative data, such as statistics or numerical measurements, to provide additional context or to support the qualitative data.

The goal of an illustrative case study is to provide a rich and detailed description of the unit being studied, and to use this information to illustrate broader themes or concepts. For example, an illustrative case study of a successful community development project may be used to illustrate the importance of community engagement and collaboration in achieving development goals.

One of the strengths of an illustrative case study is its ability to provide a detailed and nuanced understanding of a particular issue or phenomenon. By focusing on a single case, the researcher is able to provide a detailed and in-depth analysis that may not be possible through other research methods.

However, one limitation of an illustrative case study is that the findings may not be generalizable to other contexts or populations. Because the case study focuses on a single unit, it may not be representative of other similar units or situations.

A well-executed case study can shed light on wider research topics or concepts through its thorough and descriptive analysis of a specific event or phenomenon.

Exploratory Case Study

An exploratory case study is a type of case study that is used to investigate a new or previously unexplored phenomenon or issue. It is often used in research when the topic is relatively unknown or when there is little existing literature on the topic.

Exploratory case studies are typically qualitative in nature and use a variety of methods to collect data, such as interviews, observations, and document analysis. The focus of the study is to gather as much information as possible about the phenomenon being studied and to identify new and emerging themes or patterns.

The goal of an exploratory case study is to provide a foundation for further research and to generate hypotheses about the phenomenon being studied. By exploring the topic in-depth, the researcher can identify new areas of research and generate new questions to guide future research.

One of the strengths of an exploratory case study is its ability to provide a rich and detailed understanding of a new or emerging phenomenon. By using a variety of data collection methods, the researcher can gather a broad range of data and perspectives to gain a more comprehensive understanding of the phenomenon being studied.

However, one limitation of an exploratory case study is that the findings may not be generalizable to other contexts or populations. Because the study is focused on a new or previously unexplored phenomenon, the findings may not be applicable to other situations or populations.

Exploratory case studies are an effective research strategy for learning about novel occurrences, developing research hypotheses, and gaining a deep familiarity with a topic of study.

Critical Instance Case Study

A critical instance case study is a type of case study that focuses on a specific event or situation that is critical to understanding a broader issue or phenomenon. The goal of a critical instance case study is to analyze the event in depth and to draw conclusions about the broader issue or phenomenon based on the analysis.

A critical instance case study typically uses qualitative data, such as interviews, observations, or document analysis, to provide a detailed and nuanced understanding of the event being studied. The data are analyzed using various methods, such as content analysis or thematic analysis, to identify patterns and themes that emerge from the data.

The critical instance case study is often used in research when a particular event or situation is critical to understanding a broader issue or phenomenon. For example, a critical instance case study of a successful disaster response effort may be used to identify key factors that contributed to the success of the response, and to draw conclusions about effective disaster response strategies more broadly.

One of the strengths of a critical instance case study is its ability to provide a detailed and in-depth analysis of a particular event or situation. By focusing on a critical instance, the researcher is able to provide a rich and nuanced understanding of the event, and to draw conclusions about broader issues or phenomena based on the analysis.

However, one limitation of a critical instance case study is that the findings may not be generalizable to other contexts or populations. Because the case study focuses on a specific event or situation, the findings may not be applicable to other similar events or situations.

A critical instance case study is a valuable research method that can provide a detailed and nuanced understanding of a particular event or situation and can be used to draw conclusions about broader issues or phenomena based on the analysis.

Program Implementation Program Implementation

A program implementation case study is a type of case study that focuses on the implementation of a particular program or intervention. The goal of the case study is to provide a detailed and comprehensive account of the program implementation process, and to identify factors that contributed to the success or failure of the program.

Program implementation case studies typically use qualitative data, such as interviews, observations, and document analysis, to provide a detailed and nuanced understanding of the program implementation process. The data are analyzed using various methods, such as content analysis or thematic analysis, to identify patterns and themes that emerge from the data.

The program implementation case study is often used in research to evaluate the effectiveness of a particular program or intervention, and to identify strategies for improving program implementation in the future. For example, a program implementation case study of a school-based health program may be used to identify key factors that contributed to the success or failure of the program, and to make recommendations for improving program implementation in similar settings.

One of the strengths of a program implementation case study is its ability to provide a detailed and comprehensive account of the program implementation process. By using qualitative data, the researcher is able to capture the complexity and nuance of the implementation process, and to identify factors that may not be captured by quantitative data alone.

However, one limitation of a program implementation case study is that the findings may not be generalizable to other contexts or populations. Because the case study focuses on a specific program or intervention, the findings may not be applicable to other programs or interventions in different settings.

An effective research tool, a case study of program implementation may illuminate the intricacies of the implementation process and point the way towards future enhancements.

Program Effects Case Study

A program effects case study is a research method that evaluates the effectiveness of a particular program or intervention by examining its outcomes or effects. The purpose of this type of case study is to provide a detailed and comprehensive account of the program’s impact on its intended participants or target population.

A program effects case study typically employs both quantitative and qualitative data collection methods, such as surveys, interviews, and observations, to evaluate the program’s impact on the target population. The data is then analyzed using statistical and thematic analysis to identify patterns and themes that emerge from the data.

The program effects case study is often used to evaluate the success of a program and identify areas for improvement. For example, a program effects case study of a community-based HIV prevention program may evaluate the program’s effectiveness in reducing HIV transmission rates among high-risk populations and identify factors that contributed to the program’s success.

One of the strengths of a program effects case study is its ability to provide a detailed and nuanced understanding of a program’s impact on its intended participants or target population. By using both quantitative and qualitative data, the researcher can capture both the objective and subjective outcomes of the program and identify factors that may have contributed to the outcomes.

However, a limitation of the program effects case study is that it may not be generalizable to other populations or contexts. Since the case study focuses on a particular program and population, the findings may not be applicable to other programs or populations in different settings.

A program effects case study is a good way to do research because it can give a detailed look at how a program affects the people it is meant for. This kind of case study can be used to figure out what needs to be changed and how to make programs that work better.

Cumulative Case Study

A cumulative case study is a type of case study that involves the collection and analysis of multiple cases to draw broader conclusions. Unlike a single-case study, which focuses on one specific case, a cumulative case study combines multiple cases to provide a more comprehensive understanding of a phenomenon.

The purpose of a cumulative case study is to build up a body of evidence through the examination of multiple cases. The cases are typically selected to represent a range of variations or perspectives on the phenomenon of interest. Data is collected from each case using a range of methods, such as interviews, surveys, and observations.

The data is then analyzed across cases to identify common themes, patterns, and trends. The analysis may involve both qualitative and quantitative methods, such as thematic analysis and statistical analysis.

The cumulative case study is often used in research to develop and test theories about a phenomenon. For example, a cumulative case study of successful community-based health programs may be used to identify common factors that contribute to program success, and to develop a theory about effective community-based health program design.

One of the strengths of the cumulative case study is its ability to draw on a range of cases to build a more comprehensive understanding of a phenomenon. By examining multiple cases, the researcher can identify patterns and trends that may not be evident in a single case study. This allows for a more nuanced understanding of the phenomenon and helps to develop more robust theories.

However, one limitation of the cumulative case study is that it can be time-consuming and resource-intensive to collect and analyze data from multiple cases. Additionally, the selection of cases may introduce bias if the cases are not representative of the population of interest.

In summary, a cumulative case study is a valuable research method that can provide a more comprehensive understanding of a phenomenon by examining multiple cases. This type of case study is particularly useful for developing and testing theories and identifying common themes and patterns across cases.

When conducting a case study evaluation approach, one of the main challenges is the need to establish a contextually relevant research design that accounts for the unique factors of the case being studied. This requires close monitoring of the case, its environment, and relevant stakeholders. In addition, the researcher must build a framework for the collection and analysis of data that is able to draw meaningful conclusions and provide valid insights into the dynamics of the case. Ultimately, an effective case study monitoring evaluation approach will allow researchers to form an accurate understanding of their research subject.

Additionally, depending on the size and scope of the case, there may be concerns regarding the availability of resources and personnel that could be allocated to data collection and analysis. To address these issues, a case study monitoring evaluation approach can be adopted, which would involve a mix of different methods such as interviews, surveys, focus groups and document reviews. Such an approach could provide valuable insights into the effectiveness and implementation of the case in question. Additionally, this type of evaluation can be tailored to the specific needs of the case study to ensure that all relevant data is collected and respected.

When dealing with a highly sensitive or confidential subject matter within a case study, researchers must take extra measures to prevent bias during data collection as well as protect participant anonymity while also collecting valid data in order to ensure reliable results

Moreover, when conducting a case study evaluation it is important to consider the potential implications of the data gathered. By taking extra measures to prevent bias and protect participant anonymity, researchers can ensure reliable results while also collecting valid data. Maintaining confidentiality and deploying ethical research practices are essential when conducting a case study to ensure an unbiased and accurate monitoring evaluation.

When planning and implementing a case study evaluation approach, it is important to ensure the guiding principles of research quality, data collection, and analysis are met. To ensure these principles are upheld, it is essential to develop a comprehensive monitoring and evaluation plan. This plan should clearly outline the steps to be taken during the data collection and analysis process. Furthermore, the plan should provide detailed descriptions of the project objectives, target population, key indicators, and timeline. It is also important to include metrics or benchmarks to monitor progress and identify any potential areas for improvement. By implementing such an approach, it will be possible to ensure that the case study evaluation approach yields valid and reliable results.

To ensure successful implementation, it is essential to establish a reliable data collection process that includes detailed information such as the scope of the study, the participants involved, and the methods used to collect data. Additionally, it is important to have a clear understanding of what will be examined through the evaluation process and how the results will be used. All in all, it is essential to establish a sound monitoring evaluation approach for a successful case study implementation. This includes creating a reliable data collection process that encompasses the scope of the study, the participants involved, and the methods used to collect data. It is also imperative to have an understanding of what will be examined and how the results will be utilized. Ultimately, effective planning is key to ensure that the evaluation process yields meaningful insights.

Benefits of Incorporating the Case Study Evaluation Approach in the Monitoring and Evaluation of Projects and Programmes

Using a case study approach in monitoring and evaluation allows for a more detailed and in-depth exploration of the project’s success, helping to identify key areas of improvement and successes that may have been overlooked through traditional evaluation. Through this case study method, specific data can be collected and analyzed to identify trends and different perspectives that can support the evaluation process. This data can allow stakeholders to gain a better understanding of the project’s successes and failures, helping them make informed decisions on how to strengthen current activities or shape future initiatives. From a monitoring and evaluation standpoint, this approach can provide an increased level of accuracy in terms of accurately assessing the effectiveness of the project.

This can provide valuable insights into what works—and what doesn’t—when it comes to implementing projects and programs, aiding decision-makers in making future plans that better meet their objectives However, monitoring and evaluation is just one approach to assessing the success of a case study. It does provide a useful insight into what initiatives may be successful, but it is important to note that there are other effective research methods, such as surveys and interviews, that can also help to further evaluate the success of a project or program.

In conclusion, a case study evaluation approach can be incredibly useful in monitoring and evaluating complex programs and policies. By exploring key themes, patterns and relationships, organizations can gain a detailed understanding of the successes, challenges and limitations of their program or policy. This understanding can then be used to inform decision-making and improve outcomes for those involved. With its ability to provide an in-depth understanding of a program or policy, the case study evaluation approach has become an invaluable tool for monitoring and evaluation professionals.

Leave a Comment Cancel Reply

Your email address will not be published.

How strong is my Resume?

Only 2% of resumes land interviews.

Land a better, higher-paying career

Jobs for You

Cla coordinator/report officer.

• Bosnia and Herzegovina

Director of Collaborating, Learning, and Adapting (CLA) – Bosnia and Herzegovina

Senior transition and closeout consultant, global health technical and mission support (gh-tams), intern – pricing and budget.

• United States

Research Technical Advisor

• South Bend, IN, USA (Remote)
• University of Notre Dame

YMELP II Short-Term Technical Assistance (STTA)

Water, sanitation and hygiene advisor (wash) – usaid/drc.

• Democratic Republic of the Congo

Health Supply Chain Specialist – USAID/DRC

Chief of party – bosnia and herzegovina, project manager i, business development associate, director of finance and administration, request for information – collecting information on potential partners for local works evaluation.

• Washington, USA

Principal Field Monitors

Technical expert (health, wash, nutrition, education, child protection, hiv/aids, supplies), services you might be interested in, useful guides ....

How to Create a Strong Resume

Monitoring And Evaluation Specialist Resume

Resume Length for the International Development Sector

Types of Evaluation

Monitoring, Evaluation, Accountability, and Learning (MEAL)

LAND A JOB REFERRAL IN 2 WEEKS (NO ONLINE APPS!)

Sign Up & To Get My Free Referral Toolkit Now:

• Open access
• Published: 16 August 2022

Developing an implementation research logic model: using a multiple case study design to establish a worked exemplar

• Louise Czosnek   ORCID: orcid.org/0000-0002-2362-6888 1 ,
• Eva M. Zopf 1 , 2 ,
• Prue Cormie 3 , 4 ,
• Simon Rosenbaum 5 , 6 ,
• Justin Richards 7 &
• Nicole M. Rankin 8 , 9  

Implementation Science Communications volume  3 , Article number:  90 ( 2022 ) Cite this article

8903 Accesses

4 Citations

21 Altmetric

Metrics details

Implementation science frameworks explore, interpret, and evaluate different components of the implementation process. By using a program logic approach, implementation frameworks with different purposes can be combined to detail complex interactions. The Implementation Research Logic Model (IRLM) facilitates the development of causal pathways and mechanisms that enable implementation. Critical elements of the IRLM vary across different study designs, and its applicability to synthesizing findings across settings is also under-explored. The dual purpose of this study is to develop an IRLM from an implementation research study that used case study methodology and to demonstrate the utility of the IRLM to synthesize findings across case sites.

The method used in the exemplar project and the alignment of the IRLM to case study methodology are described. Cases were purposely selected using replication logic and represent organizations that have embedded exercise in routine care for people with cancer or mental illness. Four data sources were selected: semi-structured interviews with purposely selected staff, organizational document review, observations, and a survey using the Program Sustainability Assessment Tool (PSAT). Framework analysis was used, and an IRLM was produced at each case site. Similar elements within the individual IRLM were identified, extracted, and re-produced to synthesize findings across sites and represent the generalized, cross-case findings.

The IRLM was embedded within multiple stages of the study, including data collection, analysis, and reporting transparency. Between 33-44 determinants and 36-44 implementation strategies were identified at sites that informed individual IRLMs. An example of generalized findings describing “intervention adaptability” demonstrated similarities in determinant detail and mechanisms of implementation strategies across sites. However, different strategies were applied to address similar determinants. Dependent and bi-directional relationships operated along the causal pathway that influenced implementation outcomes.

Conclusions

Case study methods help address implementation research priorities, including developing causal pathways and mechanisms. Embedding the IRLM within the case study approach provided structure and added to the transparency and replicability of the study. Identifying the similar elements across sites helped synthesize findings and give a general explanation of the implementation process. Detailing the methods provides an example for replication that can build generalizable knowledge in implementation research.

Peer Review reports

Contributions to the literature

Logic models can help understand how and why evidence-based interventions (EBIs) work to produce intended outcomes.

The implementation research logic model (IRLM) provides a method to understand causal pathways, including determinants, implementation strategies, mechanisms, and implementation outcomes.

We describe an exemplar project using a multiple case study design that embeds the IRLM at multiple stages. The exemplar explains how the IRLM helped synthesize findings across sites by identifying the common elements within the causal pathway.

By detailing the exemplar methods, we offer insights into how this approach of using the IRLM is generalizable and can be replicated in other studies.

The practice of implementation aims to get “someone…, somewhere… to do something differently” [ 1 ]. Typically, this involves changing individual behaviors and organizational processes to improve the use of evidence-based interventions (EBIs). To understand this change, implementation science applies different theories, models, and frameworks (hereafter “frameworks”) to describe and evaluate the factors and steps in the implementation process [ 2 , 3 , 4 , 5 ]. Implementation science provides much-needed theoretical frameworks and a structured approach to process evaluations. One or more frameworks are often used within a program of work to investigate the different stages and elements of implementation [ 6 ]. Researchers have acknowledged that the dynamic implementation process could benefit from using logic models [ 7 ]. Logic models offer a systematic approach to combining multiple frameworks and to building causal pathways that explain the mechanisms behind individual and organizational change.

Logic models visually represent how an EBI is intended to work [ 8 ]. They link the available resources with the activities undertaken, the immediate outputs of this work, and the intermediate outcomes and longer-term impacts [ 8 , 9 ]. Through this process, causal pathways are identified. For implementation research, the causal pathway provides the interconnection between a chosen EBI, determinants, implementation strategies, and implementation outcomes [ 10 ]. Testing causal mechanisms in the research translation pathway will likely dominate the next wave of implementation research [ 11 , 12 ]. Causal mechanisms (or mechanisms of change) are the “process or event through which an implementation strategy operates to affect desired implementation outcomes” [ 13 ]. Identifying mechanisms can improve implementation strategies’ selection, prioritization, and targeting [ 12 , 13 ]. This provides an efficient and evidence-informed approach to implementation.

Implementation researchers have proposed several methods to develop and examine causal pathways [ 14 , 15 ] and mechanisms [ 16 , 17 ]. This includes formalizing the inherent relationship between frameworks via developing the Implementation Research Logic Model (IRLM) [ 7 ]. The IRLM is a logic model designed to improve the rigor and reproducibility of implementation research. It specifies the relationship between elements of implementation (determinant, strategies, and outcomes) and the mechanisms of change. To do this, it recommends linking implementation frameworks or relevant taxonomies (e.g., determinant and evaluation frameworks and implementation strategy taxonomy). The IRLM authors suggest the tool has multiple uses, including planning, executing, and reporting on the implementation process and synthesizing implementation findings across different contexts [ 7 ]. During its development, the IRLM was tested to confirm its utility in planning, executing, and reporting; however, its utility in synthesizing findings across different contexts is ongoing. Users of the tool are encouraged to consider three principles: (1) comprehensiveness in reporting determinants, implementation strategies, and implementation outcomes; (2) specifying the conceptual relationships via diagrammatic tools such as colors and arrows; and (3) detailing important elements of the study design. Further, the authors also recognize that critical elements of IRLM will vary across different study designs.

This study describes the development of an IRLM from a multiple case study design. Case study methodology can answer “how and why” questions about implementation. They enable researchers to develop a rich, in-depth understanding of a contemporary phenomenon within its natural context [ 18 , 19 , 20 , 21 ]. These methods can create coherence in the dynamic context in which EBIs exist [ 22 , 23 ]. Case studies are common in implementation research [ 24 , 25 , 26 , 27 , 28 , 29 , 30 ], with multiple case study designs suitable for undertaking comparisons across contexts [ 31 , 32 ]. However, they are infrequently applied to establish mechanisms [ 11 ] or combine implementation elements to synthesize findings across contexts (as possible through the IRLM). Hollick and colleagues [ 33 ] undertook a comparative case study, guided by a determinant framework, to explore how context influences successful implementation. The authors contrasted determinants across sites where implementation was successful versus sites where implementation failed. The study did not extend to identifying implementation strategies or mechanisms. By contrast, van Zelm et al. [ 31 ] undertook a theory-driven evaluation of successful implementation across ten hospitals. They used joint displays to present mechanisms of change aligned with evaluation outcomes; however, they did not identify the implementation strategies within the causal pathway. Our study seeks to build on these works and explore the utility of the IRLM in synthesizing findings across sites. The dual objectives of this paper were to:

Describe how case study methods can be applied to develop an IRLM

Demonstrate the utility of the IRLM in synthesizing implementation findings across case sites.

In this section, we describe the methods used in the exemplar case study and the alignment of the IRLM to this approach. The exemplar study explored the implementation of exercise EBIs in the context of the Australian healthcare system. The exemplar study aimed to investigate the integration of exercise EBIs within routine mental illness or cancer care. The evidence base detailing the therapeutic benefits of exercise for non-communicable diseases such as cancer and mental illness are extensively documented [ 34 , 35 , 36 ] but inconsistently implemented as part of routine care [ 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 ].

Additional file 1 provides the Standards for Reporting Qualitative Research (SRQR).

Case study approach

We adopted an approach to case studies based on the methods described by Yin [ 18 ]. This approach is said to have post-positivist philosophical leanings, which are typically associated with the quantitative paradigm [ 19 , 45 , 46 ]. This is evidenced by the structured, deductive approach to the methods that are described with a constant lens on objectivity, validity, and generalization [ 46 ]. Yin’s approach to case studies aligns with the IRLM for several reasons. The IRLM is designed to use established implementation frameworks. The two frameworks and one taxonomy applied in our exemplar were the Consolidated Framework for Implementation Research (CFIR) [ 47 ], Expert Recommendations for Implementing Change (ERIC) [ 48 ], and Proctor et al.’s implementation outcomes framework [ 49 ]. These frameworks guided multiple aspects of our study (see Table 1 ). Commencing an implementation study with a preconceived plan based upon established frameworks is deductive [ 22 ]. Second, the IRLM has its foundation in logic modeling to develop cause and effect relationships [ 8 ]. Yin advocates using logic models to analyze case study findings [ 18 ]. They argue that developing logic models encourages researchers to iterate and consider plausible counterfactual explanations before upholding the causal pathway. Further, Yin notes that case studies are particularly valuable for explaining the transitions and context within the cause-and-effect relationship [ 18 ]. In our exemplar, the transition was the mechanism between the implementation strategy and implementation outcome. Finally, the proposed function of IRLM to synthesize findings across sites aligns with the exemplar study that used a multiple case approach. Multiple case studies aim to develop generalizable knowledge [ 18 , 50 ].

Case study selection and boundaries

A unique feature of Yin’s approach to multiple case studies is using replication logic to select cases [ 18 ]. Cases are chosen to demonstrate similarities (literal replication) or differences for anticipated reasons (theoretical replication) [ 18 ]. In the exemplar study, the cases were purposely selected using literal replication and displayed several common characteristics. First, all cases had delivered exercise EBIs within normal operations for at least 12 months. Second, each case site delivered exercise EBIs as part of routine care for a non-communicable disease (cancer or mental illness diagnosis). Finally, each site delivered the exercise EBI within the existing governance structures of the Australian healthcare system. That is, the organizations used established funding and service delivery models of the Australian healthcare system.

Using replication logic, we posited that sites would exhibit some similarities in the implementation process across contexts (literal replication). However, based on existing implementation literature [ 32 , 51 , 52 , 53 ], we expected sites to adapt the EBIs through the implementation process. The determinant analysis should explain these adaptions, which is informed by the CFIR (theoretical replication). Finally, in case study methods, clearly defining the boundaries of each case and the units of analysis, such as individual, the organization or intervention, helps focus the research. We considered each healthcare organization as a separate case. Within that, organizational-level analysis [ 18 , 54 ] and operationalizing the implementation outcomes focused inquiry (Table 1 ).

Data collection

During the study conceptualization for the exemplar, we mapped the data sources to the different elements of the IRLM (Fig. 1 ). Four primary data sources informed data collection: (1) semi-structured interviews with staff; (2) document review (such as meeting minutes, strategic plans, and consultant reports); (3) naturalistic observations; and (4) a validated survey (Program Sustainability Assessment Tool (PSAT)). A case study database was developed using Microsoft Excel to manage and organize data collection [ 18 , 54 ].

Conceptual frame for the study

Semi-structured interviews

An interview guide was developed, informed by the CFIR interview guide tool [ 55 ]. Questions were selected across the five domains of the CFIR, which aligned with the delineation of determinant domains in the IRLM. Purposeful selection was used to identify staff for the interviews [ 56 ]. Adequate sample size in qualitative studies, particularly regarding the number of interviews, is often determined when data saturation is reached [ 57 , 58 ]. Unfortunately, there is little consensus on the definition of saturation [ 59 ], how to interpret when it has occurred [ 57 ], or whether it is possible to pre-determine in qualitative studies [ 60 ]. The number of participants in this study was determined based on the staff’s differential experience with the exercise EBI and their role in the organization. This approach sought to obtain a rounded view of how the EBI operated at each site [ 23 , 61 ]. Focusing on staff experiences also aligned with the organizational lens that bounded the study. Typical roles identified for the semi-structured interviews included the health professional delivering the EBI, the program manager responsible for the EBI, an organizational executive, referral sources, and other health professionals (e.g., nurses, allied health). Between five and ten interviews were conducted at each site. Interview times ranged from 16 to 72 min, most lasting around 40 min per participant.

Document review

A checklist informed by case study literature was developed outlining the typical documents the research team was seeking [ 18 ]. The types of documents sought to review included job descriptions, strategic plans/planning documents, operating procedures and organizational policies, communications (e.g., website, media releases, email, meeting minutes), annual reports, administrative databases/files, evaluation reports, third party consultant reports, and routinely collected numerical data that measured implementation outcomes [ 27 ]. As each document was identified, it was numbered, dated, and recorded in the case study database with a short description of the content related to the research aims and the corresponding IRLM construct. Between 24 and 33 documents were accessed at each site. A total of 116 documents were reviewed across the case sites.

Naturalistic observations

The onsite observations occurred over 1 week, wherein typical organizational operations were viewed. The research team interacted with staff, asked questions, and sought clarification of what was being observed; however, they did not disrupt the usual work routines. Observations allowed us to understand how the exercise EBI operated and contrast that with documented processes and procedures. They also provided the opportunity to observe non-verbal cues and interactions between staff. While onsite, case notes were recorded directly into the case study database [ 62 , 63 ]. Between 15 and 40 h were spent on observations per site. A total of 95 h was spent across sites on direct observations.

Program sustainability assessment tool (survey)

The PSAT is a planning and evaluation tool that assesses the sustainability of an intervention across eight domains [ 64 , 65 , 66 ]: (1) environmental support, (2) funding stability, (3) partnerships, (4) organizational capacity, (5) program evaluation, (6) program adaption, (7) communication, and (8) strategic planning [ 64 , 65 ]. The PSAT was administered to a subset of at least three participants per site who completed the semi-structured interview. The results were then pooled to provide an organization-wide view of EBI sustainability. Three participants per case site are consistent with previous studies that have used the tool [ 67 , 68 ] and recommendations for appropriate use [ 65 , 69 ].

We included a validated measure of sustainability, recognizing calls to improve understanding of this aspect of implementation [ 70 , 71 , 72 ]. Noting the limited number of measurement tools for evaluating sustainability [ 73 ], the PSAT’s characteristics displayed the best alignment with the study aims. To determine “best alignment,” we deferred to a study by Lennox and colleagues that helps researchers select suitable measurement tools based on the conceptualization of sustainability in the study [ 71 ]. The PSAT provides a multi-level view of sustainability. It is a measurement tool that can be triangulated with other implementation frameworks, such as the CFIR [ 74 ], to interrogate better and understand the later stages of implementation. Further, the tool provides a contemporary account of an EBIs capacity for sustainability [ 75 ]. This is consistent with case study methods, which explore complex, contemporary, real-life phenomena.

The voluminous data collection that is possible through case studies, and is often viewed as a challenge of the method [ 19 ], was advantageous to developing the IRLM in the exemplar and identifying the causal pathways. First, it aided three types of triangulation through the study (method, theory, and data source triangulation) [ 76 ]. Method triangulation involved collecting evidence via four methods: interview, observations, document review, and survey. Theoretical triangulation involved applying two frameworks and one taxonomy to understand and interpret the findings. Data source triangulation involved selecting participants with different roles within the organization to gain multiple perspectives about the phenomena being studied. Second, data collection facilitated depth and nuance in detailing determinants and implementation strategies. For the determinant analysis, this illuminated the subtleties within context and improved confidence and accuracy for prioritizing determinants. As case studies are essentially “naturalistic” studies, they provide insight into strategies that are implementable in pragmatic settings. Finally, the design’s flexibility enabled the integration of a survey and routinely collected numerical data as evaluation measures for implementation outcomes. This allowed us to contrast “numbers” against participants’ subjective experience of implementation [ 77 ].

Data analysis

Descriptive statistics were calculated for the PSAT and combined with the three other data sources wherein framework analysis [ 78 , 79 ] was used to analyze the data. Framework analysis includes five main phases: familiarization, identifying a thematic framework, indexing, charting, and mapping and interpretation [ 78 ]. Familiarization occurred concurrently with data collection, and the thematic frame was aligned to the two frameworks and one taxonomy we applied to the IRLM. To index and chart the data, the raw data was uploaded into NVivo 12 [ 80 ]. Codes were established to guide indexing that aligned with the thematic frame. That is, determinants within the CFIR [ 47 ], implementation strategies listed in ERIC [ 48 ], and the implementation outcomes [ 49 ] of acceptability, fidelity, penetration, and sustainability were used as codes in NVivo 12. This process produced a framework matrix that summarized the information housed under each code at each case site.

The final step of framework analysis involves mapping and interpreting the data. We used the IRLM to map and interpret the data in the exemplar. First, we identified the core elements of the implemented exercise EBI. Next, we applied the CFIR valance and strength coding to prioritize the contextual determinants. Then, we identified the implementation strategies used to address the contextual determinants. Finally, we provided a rationale (a causal mechanism) for how these strategies worked to address barriers and contribute to specific implementation outcomes. The systematic approach advocated by the IRLM provided a transparent representation of the causal pathway underpinning the implementation of the exercise EBIs. This process was followed at each case site to produce an IRLM for each organization. To compare, contrast, and synthesize findings across sites, we identified the similarities and differences in the individual IRLMs and then developed an IRLM that explained a generalized process for implementation. Through the development of the causal pathway and mechanisms, we deferred to existing literature seeking to establish these relationships [ 81 , 82 , 83 ]. Aligned with case study methods, this facilitated an iterative process of constant comparison and challenging the proposed causal relationships. Smith and colleagues advise that the IRLM “might be viewed as a somewhat simplified format,” and users are encouraged to “iterate on the design of the IRLM to increase its utility” [ 7 ]. Thus, we re-designed the IRLM within a traditional logic model structure to help make sense of the data collected through the case studies. Figure 1 depicts the conceptual frame for the study and provides a graphical representation of how the IRLM pathway was produced.

The results are presented with reference to the three principles of the IRLM: comprehensiveness, indicating the key conceptual relationship and specifying critical study design . The case study method allowed for comprehensiveness through the data collection and analysis described above. The mean number of data sources informing the analysis and development of the causal pathway at each case site was 63.75 (interviews ( M = 7), observational hours ( M =23.75), PSAT ( M =4), and document review ( M = 29). This resulted in more than 30 determinants and a similar number of implementation strategies identified at each site (determinant range per site = 33–44; implementation strategy range per site = 36–44). Developing a framework matrix meant that each determinant (prioritized and other), implementation strategy, and implementation outcome were captured. The matrix provided a direct link to the data sources that informed the content within each construct. An example from each construct was collated alongside the summary to evidence the findings.

The key conceptual relationship was articulated in a traditional linear process by aligning determinant → implementation strategy → mechanism → implementation outcome, as per the IRLM. To synthesize findings across sites, we compared and contrasted the results within each of the individual IRLM and extracted similar elements to develop a generalized IRLM that represents cross-case findings. By redeveloping the IRLM within a traditional logic model structure, we added visual representations of the bi-directional and dependent relationships, illuminating the dynamism within the implementation process. To illustrate, intervention adaptability was a prioritized determinant and enabler across sites. Healthcare providers recognized that adapting and tailoring exercise EBIs increased “fit” with consumer needs. This also extended to adapting how healthcare providers referred consumers to exercise so that it was easy in the context of their other work priorities. Successful adaption was contingent upon a qualified workforce with the required skills and competencies to enact change. Different implementation strategies were used to make adaptions across sites, such as promoting adaptability and using data experts. However, despite the different strategies, successful adaptation created positive bi-directional relationships. That is, healthcare providers’ confidence and trust in the EBI grew as consumer engagement increased and clinical improvements were observed. This triggered greater engagement with the EBI (e.g., acceptability → penetration → sustainability), albeit the degree of engagement differed across sites. Figure 2 illustrates this relationship within the IRLM and provides a contrasting relationship by highlighting how a prioritized barrier across sites (available resources) was addressed.

Example of intervention adaptability (E) contrasted with available resources (B) within a synthesised IRLM across case sites

The final principle is to specify critical study design , wherein we have described how case study methodology was used to develop the IRLM exemplar. Our intention was to produce an explanatory causal pathway for the implementation process. The implementation outcomes of acceptability and fidelity were measured at the level of the provider, and penetration and sustainability were measured at the organizational level [ 49 ]. Service level and clinical level outcomes were not identified for a priori measurement throughout the study. We did identify evidence of clinical outcomes that supported our overall findings via the document review. Historical evaluations on the service indicated patients increased their exercise level or demonstrated a change in symptomology/function. The implementation strategies specified in the study were those chosen by the organizations. We did not attempt to augment routine practice or change implementation outcomes by introducing new strategies. The barriers across sites were represented with a (B) symbol and enablers with an (E) symbol in the IRLM. In the individual IRLM, consistent determinants and strategies were highlighted (via bolding) to support extraction. Finally, within the generalized IRLM, the implementation strategies are grouped according to the ERIC taxonomy category. This accounts for the different strategies applied to achieve similar outcomes across case studies.

This study provides a comprehensive overview that uses case study methodology to develop an IRLM in an implementation research project. Using an exemplar that examines implementation in different healthcare settings, we illustrate how the IRLM (that documents the causal pathways and mechanisms) was developed and enabled the synthesis of findings across sites.

Case study methodologies are fraught with inconsistencies in terminology and approach. We adopted the method described by Yin. Its guiding paradigm, which is rooted in objectivity, means it can be viewed as less flexible than other approaches [ 46 , 84 ]. We found the approach offered sufficient flexibility within the frame of a defined process. We argue that the defined process adds to the rigor and reproducibility of the study, which is consistent with the principles of implementation science. That is, accessing multiple sources of evidence, applying replication logic to select cases, maintaining a case study database, and developing logic models to establish causal pathways, demonstrates the reliability and validity of the study. The method was flexible enough to embed the IRLM within multiple phases of the study design, including conceptualization, philosophical alignment, and analysis. Paparini and colleagues [ 85 ] are developing guidance that recognizes the challenges and unmet value of case study methods for implementation research. This work, supported by the UK Medical Research Council, aims to enhance the conceptualization, application, analysis, and reporting of case studies. This should encourage and support researchers to use case study methods in implementation research with increased confidence.

The IRLM produced a relatively linear depiction of the relationship between context, strategies, and outcomes in our exemplar. However, as noted by the authors of the IRLM, the implementation process is rarely linear. If the tool is applied too rigidly, it may inadvertently depict an overly simplistic view of a complex process. To address this, we redeveloped the IRLM within a traditional logic model structure, adding visual representations of the dependent and bidirectional relationships evident within the general IRLM pathway [ 86 ]. Further, developing a general IRLM of cross-case findings that synthesized results involved a more inductive approach to identifying and extracting similar elements. It required the research team to consider broader patterns in the data before offering a prospective account of the implementation process. This was in contrast to the earlier analysis phases that directly mapped determinants and strategies to the CFIR and ERIC taxonomy. We argue that extracting similar elements is analogous to approaches that have variously been described as portable elements [ 87 ], common elements [ 88 ], or generalization by mechanism [ 89 ]. While defined and approached slightly differently, these approaches aim to identify elements frequently shared across effective EBIs and thus can form the basis of future EBIs to increase their utility, efficiency, and effectiveness [ 88 ]. We identified similarities related to determinant detail and mechanism of different implementation strategies across sites. This finding supports the view that many implementation strategies could be suitable, and selecting the “right mix” is challenging [ 16 ]. Identifying common mechanisms, such as increased motivation, skill acquisition, or optimizing workflow, enabled elucidation of the important functions of strategies. This can help inform the selection of appropriate strategies in future implementation efforts.

Finally, by developing individual IRLMs and then re-producing a general IRLM, we synthesized findings across sites and offered generalized findings. The ability to generalize from case studies is debated [ 89 , 90 ], with some considering the concept a fallacy [ 91 ]. That is, the purpose of qualitative research is to develop a richness through data that is situated within a unique context. Trying to extrapolate from findings is at odds with exploring unique context. We suggest the method described herein and the application of IRLM could be best applied to a form of generalization called ‘transferability’ [ 91 , 92 ]. This suggests that findings from one study can be transferred to another setting or population group. In this approach, the new site takes the information supplied and determines those aspects that would fit with their unique environment. We argue that elucidating the implementation process across multiple sites improves the confidence with which certain “elements” could be applied to future implementation efforts. For example, our approach may also be helpful for multi-site implementation studies that use methods other than case studies. Developing a general IRLM through study conceptualization could identify consistencies in baseline implementation status across sites. Multi-site implementation projects may seek to introduce and empirically test implementation strategies, such as via a cluster randomized controlled trial [ 93 ]. Within this study design, baseline comparison between control and intervention sites might extend to a comparison of organizational type, location and size, and individual characteristics, but not the chosen implementation strategies [ 94 ]. Applying the approach described within our study could enhance our understanding of how to support effective implementation.

Limitations

After the research team conceived this study, the authors of the PSAT validated another tool for use in clinical settings (Clinical Sustainability Assessment Tool (CSAT)) [ 95 ]. This tool appears to align better with our study design due to its explicit focus on maintaining structured clinical care practices. The use of multiple data sources and consistency in some elements across the PSAT and CSAT should minimize the limitations in using the PSAT survey tool. At most case sites, limited staff were involved in developing and implementing exercise EBI. Participants who self-selected for interviews may be more invested in assuring positive outcomes for the exercise EBI. Inviting participants from various roles was intended to reduce selection bias. Finally, we recognize recent correspondence suggesting the IRLM misses a critical step in the causal pathway. That is the mechanism between determinant and selection of an appropriate implementation strategy [ 96 ]. Similarly, Lewis and colleagues note that additional elements, including pre-conditions, moderators, and mediators (distal and proximal), exist within the causal pathway [ 13 ]. Through the iterative process of developing the IRLM, decisions were made about the determinant → implementation strategy relationship; however, this is not captured in the IRLM. Secondary analysis of the case study data would allow elucidation of these relationships, as this information can be extracted through the case study database. This was outside the scope of the exemplar study.

Developing an IRLM via case study methods proved useful in identifying causal pathways and mechanisms. The IRLM can complement and enhance the study design by providing a consistent and structured approach. In detailing our approach, we offer an example of how multiple case study designs that embed the IRLM can aid the synthesis of findings across sites. It also provides a method that can be replicated in future studies. Such transparency adds to the quality, reliability, and validity of implementation research.

Availability of data and materials

The data that support the findings of this study are available on request from the corresponding author [LC]. The data are not publicly available due to them containing information that could compromise research participant privacy.

Presseau J, McCleary N, Lorencatto F, Patey AM, Grimshaw JM, Francis JJ. Action, actor, context, target, time (AACTT): a framework for specifying behaviour. Implement Sci. 2019;14(1):102.

Article   PubMed   PubMed Central   Google Scholar  

Damschroder LJ. Clarity out of chaos: use of theory in implementation research. Psychiatry Res. 2020;283(112461).

Bauer M, Damschroder L, Hagedorn H, Smith J, Kilbourne A. An introduction to implementation science for the non-specialist. BMC Psychol. 2015;3(1):32.

Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10(1):53.

Lynch EA, Mudge A, Knowles S, Kitson AL, Hunter SC, Harvey G. “There is nothing so practical as a good theory”: a pragmatic guide for selecting theoretical approaches for implementation projects. BMC Health Serv Res. 2018;18(1):857.

Birken SA, Powell BJ, Presseau J, Kirk MA, Lorencatto F, Gould NJ, et al. Combined use of the Consolidated Framework for Implementation Research (CFIR) and the Theoretical Domains Framework (TDF): a systematic review. Implement Sci. 2017;12(1):2.

Smith JD, Li DH, Rafferty MR. The Implementation Research Logic Model: a method for planning, executing, reporting, and synthesizing implementation projects. Implement Sci. 2020;15(1):84.

Kellogg WK. Foundation. Logic model development guide. Michigan, USA; 2004.

McLaughlin JA, Jordan GB. Logic models: a tool for telling your programs performance story. Eval Prog Plann. 1999;22(1):65–72.

Article   Google Scholar  

Anselmi L, Binyaruka P, Borghi J. Understanding causal pathways within health systems policy evaluation through mediation analysis: an application to payment for performance (P4P) in Tanzania. Implement Sci. 2017;12(1):10.

Lewis C, Boyd M, Walsh-Bailey C, Lyon A, Beidas R, Mittman B, et al. A systematic review of empirical studies examining mechanisms of implementation in health. Implement Sci. 2020;15(1):21.

Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, et al. Enhancing the impact of implementation strategies in healthcare: a research agenda. Front Public Health. 2019;7(3).

Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, Walsh-Bailey C and Weiner B. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. 2018;6(136).

Bartholomew L, Parcel G, Kok G. Intervention mapping: a process for developing theory and evidence-based health education programs. Health Educ Behav. 1998;25(5):545–63.

Article   CAS   PubMed   Google Scholar  

Weiner BJ, Lewis MA, Clauser SB, Stitzenberg KB. In search of synergy: strategies for combining interventions at multiple levels. JNCI Monographs. 2012;2012(44):34–41.

Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen J, Proctor EK, et al. Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. 2017;44(2):177–94.

Fernandez ME, ten Hoor GA, van Lieshout S, Rodriguez SA, Beidas RS, Parcel G, Ruiter R, Markham C and Kok G. Implementation mapping: using intervention mapping to develop implementation strategies. Frontiers. Public Health. 2019;7(158).

Yin R. Case study research and applications design and methods. 6th Edition ed. United States of America: Sage Publications; 2018.

Google Scholar  

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol. 2011;11:100.

Stake R. The art of case study reseach. United States of America: Sage Publications; 2005.

Thomas G. How to do your case study. 2nd Edition ed. London: Sage Publications; 2016.

Ramanadhan S, Revette AC, Lee RM and Aveling E. Pragmatic approaches to analyzing qualitative data for implementation science: an introduction. Implement Sci Commun. 2021;2(70).

National Cancer Institute. Qualitative methods in implementation science United States of America: National Institutes of Health Services; 2018.

Mathers J, Taylor R, Parry J. The challenge of implementing peer-led interventions in a professionalized health service: a case study of the national health trainers service in England. Milbank Q. 2014;92(4):725–53.

Powell BJ, Proctor EK, Glisson CA, Kohl PL, Raghavan R, Brownson RC, et al. A mixed methods multiple case study of implementation as usual in children’s social service organizations: study protocol. Implement Sci. 2013;8(1):92.

van de Glind IM, Heinen MM, Evers AW, Wensing M, van Achterberg T. Factors influencing the implementation of a lifestyle counseling program in patients with venous leg ulcers: a multiple case study. Implement Sci. 2012;7(1):104.

Greenhalgh T, Macfarlane F, Barton-Sweeney C, Woodard F. “If we build it, will it stay?” A case study of the sustainability of whole-system change in London. Milbank Q. 2012;90(3):516–47.

Urquhart R, Kendell C, Geldenhuys L, Ross A, Rajaraman M, Folkes A, et al. The role of scientific evidence in decisions to adopt complex innovations in cancer care settings: a multiple case study in Nova Scotia, Canada. Implement Sci. 2019;14(1):14.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Herinckx H, Kerlinger A, Cellarius K. Statewide implementation of high-fidelity recovery-oriented ACT: A case study. Implement Res Pract. 2021;2:2633489521994938.

Young AM, Hickman I, Campbell K, Wilkinson SA. Implementation science for dietitians: The ‘what, why and how’ using multiple case studies. Nutr Diet. 2021;78(3):276–85.

Article   PubMed   Google Scholar  

van Zelm R, Coeckelberghs E, Sermeus W, Wolthuis A, Bruyneel L, Panella M, et al. A mixed methods multiple case study to evaluate the implementation of a care pathway for colorectal cancer surgery using extended normalization process theory. BMC Health Serv Res. 2021;21(1):11.

Albers B, Shlonsky A, Mildon R. Implementation Science 3.0. Switzerland: Springer; 2020.

Book   Google Scholar  

Hollick RJ, Black AJ, Reid DM, McKee L. Shaping innovation and coordination of healthcare delivery across boundaries and borders. J Health Organ Manag. 2019;33(7/8):849–68.

Article   PubMed Central   Google Scholar  

Pedersen B, Saltin B. Exercise as medicine – evidence for prescribing exercise as therapy in 26 different chronic diseases. Scand J Med Sci Sports. 2015;25:1–72.

Firth J, Siddiqi N, Koyanagi A, Siskind D, Rosenbaum S, Galletly C, et al. The Lancet Psychiatry Commission: a blueprint for protecting physical health in people with mental illness. Lancet Psychiatry. 2019;6(8):675–712.

Campbell K, Winters-Stone K, Wisekemann J, May A, Schwartz A, Courneya K, et al. Exercise guidelines for cancer survivors: consensus statement from international multidisciplinary roundtable. Med Sci Sports Exerc. 2019;51(11):2375–90.

Deenik J, Czosnek L, Teasdale SB, Stubbs B, Firth J, Schuch FB, et al. From impact factors to real impact: translating evidence on lifestyle interventions into routine mental health care. Transl Behav Med. 2020;10(4):1070–3.

Suetani S, Rosenbaum S, Scott JG, Curtis J, Ward PB. Bridging the gap: What have we done and what more can we do to reduce the burden of avoidable death in people with psychotic illness? Epidemiol Psychiatric Sci. 2016;25(3):205–10.

Article   CAS   Google Scholar  

Stanton R, Rosenbaum S, Kalucy M, Reaburn P, Happell B. A call to action: exercise as treatment for patients with mental illness. Aust J Primary Health. 2015;21(2):120–5.

Rosenbaum S, Hobson-Powell A, Davison K, Stanton R, Craft LL, Duncan M, et al. The role of sport, exercise, and physical activity in closing the life expectancy gap for people with mental illness: an international consensus statement by Exercise and Sports Science Australia, American College of Sports Medicine, British Association of Sport and Exercise Science, and Sport and Exercise Science New Zealand. Transll J Am Coll Sports Med. 2018;3(10):72–3.

Chambers D, Vinson C, Norton W. Advancing the science of implementation across the cancer continuum. United States of America: Oxford University Press Inc; 2018.

Schmitz K, Campbell A, Stuiver M, Pinto B, Schwartz A, Morris G, et al. Exercise is medicine in oncology: engaging clinicians to help patients move through cancer. Cancer J Clin. 2019;69(6):468–84.

Santa Mina D, Alibhai S, Matthew A, Guglietti C, Steele J, Trachtenberg J, et al. Exercise in clinical cancer care: a call to action and program development description. Curr Oncol. 2012;19(3):9.

Czosnek L, Rankin N, Zopf E, Richards J, Rosenbaum S, Cormie P. Implementing exercise in healthcare settings: the potential of implementation science. Sports Med. 2020;50(1):1–14.

Harrison H, Birks M, Franklin R, Mills J. Case study research: foundations and methodological orientations. Forum: Qualitative. Soc Res. 2017;18(1).

Yazan B. Three approaches to case study methods in education: Yin, Merriam, and Stake. Qual Rep. 2015;20(2):134–52.

Damschroder L, Aaron D, Keith R, Kirsh S, Alexander J, Lowery J. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10(1):21.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Admin Pol Ment Health. 2011;38(2):65–76.

Heale R, Twycross A. What is a case study? Evid Based Nurs. 2018;21(1):7–8.

Brownson R, Colditz G, Proctor E. Dissemination and implementation research in health: translating science to practice. Second ed. New York: Oxford University Press; 2017.

Quiñones MM, Lombard-Newell J, Sharp D, Way V, Cross W. Case study of an adaptation and implementation of a Diabetes Prevention Program for individuals with serious mental illness. Transl Behav Med. 2018;8(2):195–203.

Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. 2019;14(1):58.

Baxter P, Jack S. Qualitative case study methodology: study design and implementation for novice researchers. Qual Rep. 2008;13(4):544–59.

Consolidated Framework for Implementation Research 2018. Available from: http://www.cfirguide.org/index.html . Cited 2018 14 February.

Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Admin Pol Ment Health. 2015;42(5):533–44.

Francis JJ, Johnston M, Robertson C, Glidewell L, Entwistle V, Eccles MP, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health. 2010;25(10):1229–45.

Teddlie C, Yu F. Mixed methods sampling: a typology with examples. J Mixed Methods Res. 2007;1(1):77–100.

Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant. 2018;52(4):1893–907.

Braun V, Clarke V. To saturate or not to saturate? Questioning data saturation as a useful concept for thematic analysis and sample-size rationales. Qual Res Sport Exerc Health. 2021;13(2):201–16.

Burau V, Carstensen K, Fredens M, Kousgaard MB. Exploring drivers and challenges in implementation of health promotion in community mental health services: a qualitative multi-site case study using Normalization Process Theory. BMC Health Serv Res. 2018;18(1):36.

Phillippi J, Lauderdale J. A guide to field notes for qualitative research: context and conversation. Qual Health Res. 2018;28(3):381–8.

Mulhall A. In the field: notes on observation in qualitative research. J Adv Nurs. 2003;41(3):306–13.

Schell SF, Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB, et al. Public health program capacity for sustainability: a new framework. Implement Sci. 2013;8(1):15.

Washington University. The Program Sustainability Assessment Tool St Louis: Washington University; 2018. Available from: https://sustaintool.org/ . Cited 2018 14 February.

Luke DA, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The Program Sustainability Assessment Tool: a new instrument for public health programs. Prev Chronic Dis. 2014;11:E12.

Stoll S, Janevic M, Lara M, Ramos-Valencia G, Stephens TB, Persky V, et al. A mixed-method application of the Program Sustainability Assessment Tool to evaluate the sustainability of 4 pediatric asthma care coordination programs. Prev Chronic Dis. 2015;12:E214.

Kelly C, Scharff D, LaRose J, Dougherty NL, Hessel AS, Brownson RC. A tool for rating chronic disease prevention and public health interventions. Prev Chronic Dis. 2013;10:E206.

Calhoun A, Mainor A, Moreland-Russell S, Maier RC, Brossart L, Luke DA. Using the Program Sustainability Assessment Tool to assess and plan for sustainability. Prev Chronic Dis. 2014;11:E11.

Proctor E, Luke D, Calhoun A, McMillen C, Brownson R, McCrary S, et al. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Implement Sci. 2015;10(1):88.

Lennox L, Maher L, Reed J. Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare. Implement Sci. 2018;13(1):27.

Moore JE, Mascarenhas A, Bain J, Straus SE. Developing a comprehensive definition of sustainability. Implement Sci. 2017;12(1):110.

Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015;10(1):155.

Shelton RC, Chambers DA, Glasgow RE. An extension of RE-AIM to enhance sustainability: addressing dynamic context and promoting health equity over time. Front Public Health. 2020;8(134).

Moullin JC, Sklar M, Green A, Dickson KS, Stadnick NA, Reeder K, et al. Advancing the pragmatic measurement of sustainment: a narrative review of measures. Implement Sci Commun. 2020;1(1):76.

Denzin N. The research act: A theoretical introduction to sociological methods. New Jersey: Transaction Publishers; 1970.

Grant BM, Giddings LS. Making sense of methodologies: a paradigm framework for the novice researcher. Contemp Nurse. 2002;13(1):10–28.

Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13(1):117.

Pope C, Ziebland S, Mays N. Qualitative research in health care. Analysing qualitative data. BMJ. 2000;320(7227):114–6.

QSR International. NVivo 11 Pro for Windows 2018. Available from: https://www.qsrinternational.com/nvivo-qualitative-data-analysissoftware/home .

Waltz TJ, Powell BJ, Fernández ME, Abadie B, Damschroder LJ. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implement Sci. 2019;14(1):42.

Michie S, Johnston M, Rothman AJ, de Bruin M, Kelly MP, Carey RN, et al. Developing an evidence-based online method of linking behaviour change techniques and theoretical mechanisms of action: a multiple methods study. Southampton (UK): NIHR Journals. Library. 2021;9:1.

Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005;14(1):26–33.

Ebneyamini S, Sadeghi Moghadam MR. Toward developing a framework for conducting case study research. Int J Qual Methods. 2018;17(1):1609406918817954.

Paparini S, Green J, Papoutsi C, Murdoch J, Petticrew M, Greenhalgh T, et al. Case study research for better evaluations of complex interventions: rationale and challenges. BMC Med. 2020;18(1):301.

Sarkies M, Long JC, Pomare C, Wu W, Clay-Williams R, Nguyen HM, et al. Avoiding unnecessary hospitalisation for patients with chronic conditions: a systematic review of implementation determinants for hospital avoidance programmes. Implement Sci. 2020;15(1):91.

Koorts H, Cassar S, Salmon J, Lawrence M, Salmon P, Dorling H. Mechanisms of scaling up: combining a realist perspective and systems analysis to understand successfully scaled interventions. Int J Behav Nutr Phys Act. 2021;18(1):42.

Engell T, Kirkøen B, Hammerstrøm KT, Kornør H, Ludvigsen KH, Hagen KA. Common elements of practice, process and implementation in out-of-school-time academic interventions for at-risk children: a systematic review. Prev Sci. 2020;21(4):545–56.

Bengtsson B, Hertting N. Generalization by mechanism: thin rationality and ideal-type analysis in case study research. Philos Soc Sci. 2014;44(6):707–32.

Tsang EWK. Generalizing from research findings: the merits of case studies. Int J Manag Rev. 2014;16(4):369–83.

Polit DF, Beck CT. Generalization in quantitative and qualitative research: myths and strategies. Int J Nurs Stud. 2010;47(11):1451–8.

Adler C, Hirsch Hadorn G, Breu T, Wiesmann U, Pohl C. Conceptualizing the transfer of knowledge across cases in transdisciplinary research. Sustain Sci. 2018;13(1):179–90.

Wolfenden L, Foy R, Presseau J, Grimshaw JM, Ivers NM, Powell BJ, et al. Designing and undertaking randomised implementation trials: guide for researchers. BMJ. 2021;372:m3721.

Nathan N, Hall A, McCarthy N, Sutherland R, Wiggers J, Bauman AE, et al. Multi-strategy intervention increases school implementation and maintenance of a mandatory physical activity policy: outcomes of a cluster randomised controlled trial. Br J Sports Med. 2022;56(7):385–93.

Malone S, Prewitt K, Hackett R, Lin JC, McKay V, Walsh-Bailey C, et al. The Clinical Sustainability Assessment Tool: measuring organizational capacity to promote sustainability in healthcare. Implement Sci Commun. 2021;2(1):77.

Sales AE, Barnaby DP, Rentes VC. Letter to the editor on “the implementation research logic model: a method for planning, executing, reporting, and synthesizing implementation projects” (Smith JD, Li DH, Rafferty MR. the implementation research logic model: a method for planning, executing, reporting, and synthesizing implementation projects. Implement Sci. 2020;15 (1):84. Doi:10.1186/s13012-020-01041-8). Implement Sci. 2021;16(1):97.

Download references

Acknowledgements

The authors would like to acknowledge the healthcare organizations and staff who supported the study.

SR is funded by an NHMRC Early Career Fellowship (APP1123336). The funding body had no role in the study design, data collection, data analysis, interpretation, or manuscript development.

Author information

Authors and affiliations.

Mary MacKillop Institute for Health Research, Australian Catholic University, Melbourne, Australia

Louise Czosnek & Eva M. Zopf

Cabrini Cancer Institute, The Szalmuk Family Department of Medical Oncology, Cabrini Health, Melbourne, Australia

Eva M. Zopf

Peter MacCallum Cancer Centre, Melbourne, Australia

Prue Cormie

Sir Peter MacCallum Department of Oncology, The University of Melbourne, Melbourne, Australia

Discipline of Psychiatry and Mental Health, University of New South Wales, Sydney, Australia

Simon Rosenbaum

School of Health Sciences, University of New South Wales, Sydney, Australia

Faculty of Health, Victoria University of Wellington, Wellington, New Zealand

Justin Richards

Faculty of Medicine and Health, University of Sydney, Sydney, Australia

Nicole M. Rankin

Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Melbourne, Australia

You can also search for this author in PubMed   Google Scholar

Contributions

LC, EZ, SR, JR, PC, and NR contributed to the conceptualization of the study. LC undertook the data collection, and LC, EZ, SR, JR, PC, and NR supported the analysis. The first draft of the manuscript was written by LC with NR and EZ providing first review. LC, EZ, SR, JR, PC, and NR commented on previous versions of the manuscript and provided critical review. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Louise Czosnek .

Ethics declarations

Ethics approval and consent to participate.

This study is approved by Sydney Local Health District Human Research Ethics Committee - Concord Repatriation General Hospital (2019/ETH11806). Ethical approval is also supplied by Australian Catholic University (2018-279E), Peter MacCallum Cancer Centre (19/175), North Sydney Local Health District - Macquarie Hospital (2019/STE14595), and Alfred Health (516-19).

Consent for publication

Not applicable.

Competing interests

PC is the recipient of a Victorian Government Mid-Career Research Fellowship through the Victorian Cancer Agency. PC is the Founder and Director of EX-MED Cancer Ltd, a not-for-profit organization that provides exercise medicine services to people with cancer. PC is the Director of Exercise Oncology EDU Pty Ltd, a company that provides fee for service training courses to upskill exercise professionals in delivering exercise to people with cancer.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Standards for Reporting Qualitative Research (SRQR).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Czosnek, L., Zopf, E.M., Cormie, P. et al. Developing an implementation research logic model: using a multiple case study design to establish a worked exemplar. Implement Sci Commun 3 , 90 (2022). https://doi.org/10.1186/s43058-022-00337-8

Download citation

Received : 19 March 2022

Accepted : 01 August 2022

Published : 16 August 2022

DOI : https://doi.org/10.1186/s43058-022-00337-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Logic model
• Case study methods
• Causal pathways
• Causal mechanisms

Implementation Science Communications

ISSN: 2662-2211

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Product overview
• All features
• App integrations

CAPABILITIES

• project icon Project management
• Project views
• Custom fields
• Status updates
• goal icon Goals and reporting
• Reporting dashboards
• workflow icon Workflows and automation
• portfolio icon Resource management
• Time tracking
• my-task icon Admin and security
• Admin console
• asana-intelligence icon Asana Intelligence
• list icon Personal
• premium icon Starter
• briefcase icon Advanced
• Goal management
• Organizational planning
• Campaign management
• Creative production
• Content calendars
• Marketing strategic planning
• Resource planning
• Project intake
• Product launches
• Employee onboarding
• View all uses arrow-right icon
• Project plans
• Team goals & objectives
• Team continuity
• Meeting agenda
• View all templates arrow-right icon
• Work management resources Discover best practices, watch webinars, get insights
• What's new Learn about the latest and greatest from Asana
• Customer stories See how the world's best organizations drive work innovation with Asana
• Help Center Get lots of tips, tricks, and advice to get the most from Asana
• Asana Academy Sign up for interactive courses and webinars to learn Asana
• Developers Learn more about building apps on the Asana platform
• Community programs Connect with and learn from Asana customers around the world
• Events Find out about upcoming events near you
• Partners Learn more about our partner programs
• Support Need help? Contact the Asana support team
• Asana for nonprofits Get more information on our nonprofit discount program, and apply.

Featured Reads

• Project planning |
• What is an implementation plan? 6 steps ...

What is an implementation plan? 6 steps to create one

An implementation plan—also known as a strategic plan—outlines the steps your team should take when accomplishing a shared goal or objective. This plan combines strategy, process, and action and will include all parts of the project from scope to budget and beyond. In this guide, we’ll discuss what an implementation plan is and how to create one.

Projects require planning to be successful. Would you build a house without a blueprint? Probably not, because nailing pieces of wood together without a plan could lead to disaster. The same concept is true in the corporate world. An implementation plan functions as the blueprint for any shared objective. Your plan should include everything from the project strategy, to the budget, to the list of people working on the project. 

In this guide, we’ll discuss what an implementation plan is and how to create one. These steps can help you and your team prepare for projects both big and small.

What is the purpose of an implementation plan?

The purpose of an implementation plan is to ensure that your team can answer the who, what, when, how, and why of a project before moving into the execution phase. In simple terms, it's the action plan that turns your strategy into specific tasks.

A good way to know whether your implementation plan is effective is to hand it to someone outside of your team and see if they can understand the project in its entirety. Your implementation plan should leave no questions unanswered.

How to create an implementation plan in 6 steps

If you want your implementation plan to be comprehensive and beneficial to your project team, you’ll need to follow specific steps and include the right components. Use the following steps when creating your plan to reduce the risk of gaps in your strategy.

1. Define goals

The first step in the implementation process is defining your goals . Determine what you hope to accomplish when your project is complete, like whether you hope to win over a new marketing client or revamp your internal content strategy. Starting with your project objectives in mind can help flesh out your project plan. 

Tips to consider:

Ask questions: When defining your goals, you and your team may want to ask questions about your project such as, “What are we trying to achieve with this project? What deliverables do we hope to produce? Who are the stakeholders we plan to share our project deliverables with?”

Brainstorm risk scenarios: Although you’ll perform a more in-depth risk assessment later on in your implementation plan, brainstorming potential risk scenarios early on gives you a more realistic idea of what you’re able to achieve. 

2. Conduct research

Once you have a broad idea of the project goals you want to achieve, you can hone in on these goals by conducting research such as interviews, surveys, focus groups, or observations. Your research should come from key experts in your field. These experts may be team members or external stakeholders. Your research outcomes should include a list of what your project timeline, budget, and personnel may look like.

Collaborate using shared tools: Collaboration is easier when you have the right communication tools in place to do so. Use a team collaboration tool to share your project goals and get feedback from others, regardless of their location. 

3. Map out risks

You brainstormed risk scenarios in step one of your implementation strategy, and in step three, you’ll map out all the potential risks you may face in your project. Risks can include anything from paid time off and holidays to budget constraints and loss of personnel. 

A great way to map out your risks is by using a risk register. This tool will help you prioritize project risks and prepare for them accordingly. You can also conduct a SWOT analysis , which will identify any weaknesses or threats affecting your project. 

Be flexible and proactive: Mapping out risks is more than just a preparation strategy. If you identify preventable risks during this stage of the implementation plan, you can take action to prevent those risks. This may mean adjusting your initial project goals. 

4. Schedule milestones

Scheduling your project milestones is an important step in the planning process because these checkpoints help you track your progress during execution. Milestones serve as metrics—they are a way to measure how far you’ve come in your project and how far you have left to go. 

To visualize project milestones and keep your entire team on track, use a Gantt chart . With a Gantt chart, you can visually lay out your implementation schedule and show how long you think each task will take.

Add wiggle room: Things don’t always go as planned, even if you do everything in your power to a schedule. By adding wiggle room to your schedule, you can ensure your project stays on track instead of keeping tight milestones and failing to meet them.

Clarify dependencies: Dependencies are tasks that rely on the completion of other tasks. Clarifying your dependencies makes it easier to keep the project on track and hit your milestones.

5. Assign responsibilities and tasks

Every action plan must include a list of responsibilities with team members assigned to each one. By assigning responsibilities, you can assess the performance of each team member and monitor progress more closely. Using a RACI chart can be an effective project management tool for clarifying roles and responsibilities. 

Assigning responsibilities is different from assigning individual tasks. One team member may be responsible for overseeing the project review, while you may assign three other team members to handle the delivery and communication of the project to various teams for review. When you assign responsibilities and tasks, be sure to make your expectations clear. 

Communication is key: When you assign roles, responsibilities, or tasks, it’s best to communicate why you’re choosing one team member over another. Instead of letting team members question why they have specific roles, you can use this step in the planning process as an opportunity to highlight team member strengths.

Track responsibilities in a shared tool: Having a shared tool, like project management software, can give team members clarity on who's doing what and by when.

6. Allocate resources

Resource allocation is one of the best ways to reduce risk. If you can plan out what resources you need for your project and ensure those resources will be available, you’ll avoid the risk of running out of resources mid-project. If you notice that you don’t have enough resources in this step of the implementation process, you can adjust your project accordingly before it kicks off. 

Resources may include money, personnel, software, equipment, and other physical or technical materials. Time can also be a resource because the team members you need to complete the project may be working on other projects.

Tips to consider: Ask yourself the following questions when identifying available resources for your project: 

What is the project’s priority level? 

Who is available to work on this project? 

What budget or tools are available? 

What additional resources do we need? 

Who needs to approve the resource allocation plan?

Following these steps as you create your implementation plan will increase the likelihood of hitting your project goals. Having a checklist of the items to include in your implementation plan can also lead to successful implementation. 

What to include in an implementation plan

Knowing how to create your implementation plan is crucial, but you also need to know what to include in your plan. This checklist includes the six most important items you’ll want to consider if you want to move forward with a successful project. 

1. Objectives

You’ll outline your project objectives in step one of the implementation process. Set your goals and decide what metrics your team will use to measure to monitor progress. By clearly identifying your project objectives, you and your team can measure progress and performance as you move forward.

2. Scope statement

You’ll set the scope of your project in step two when conducting research. Your project scope statement should outline the boundaries you’ve set for your project and broadly define what goals, deadlines, and project outcomes you’ll be working toward. Defining your project scope in the implementation plan can help prevent scope creep when you’re farther along in the project.

3. Outline of deliverables

Deliverables are the tangible goals of your project. Outlining the deliverables you hope to create can serve as a resource when managing time frames, delegating tasks, and allocating resources. 

4. Task due dates

Although the project timeline may change as your project progresses, it’s important to clarify your expected due dates during implementation planning. When you estimate task due dates, you can schedule milestones around these due dates and plan for project completion. You will commonly see Gantt charts used for strategic planning and implementation planning. This is because Gantt charts display information in a follows a linear path, similar to a timeline. 

5. Risk assessment

You’ll conduct your risk assessment in step three of the implementation process. Whether you use a   risk register , SWOT analysis , or contingency plan to identify risks , be sure to include these documents in your plan. That way, others involved in the project can look through your findings and potentially help you prevent these risks. 

6. Team member roles and responsibilities

You assigned roles and responsibilities to team members in step five of your plan, and keeping a detailed record of what these are can hold everyone accountable. Whether you use a RACI chart or another tool to clarify team member roles, there should be a place in your plan for everyone to refer to in case questions arise. 

Your implementation plan will likely be unique to the project you're working on, so it may include other components not listed above. However, you can use the six items above as your guide so you know your plan is comprehensive.

Many aspects of project implementation overlap with strategic planning. As a project manager , working on the project implementation plan while you are also working on the strategic plan can help minimize the total time spent on planning.

Another way to save time during the planning process is to house all of your plans in a work management platform. When your project team is ready to start the implementation process, everything is in one convenient place.

Benefits of having an implementation plan

There are many benefits to implementation planning, with the top benefit being an increased chance of project success. Implementing a project plan creates a roadmap for executing your project so you can prevent issues from occurring. 

Other benefits to having an implementation plan include:

Improved communication between team members and key stakeholders

Better organization and management of resources

Increased accountability for everyone involved in the project

More structured project timeline and daily workflow

Easier collaboration between team members

Going straight into the execution phase without an implementation plan may feel like walking on stage to give a speech without knowing what you’re going to say. Preparation is key for top-notch performance. 

Simplify implementation planning

Knowing the steps for implementation planning is the foundation of project management. A well-planned project leads to a successful project.

Related resources

How to use a feasibility study in project management

How to track utilization rate and drive team profitability

How to accomplish big things with long-term goals

Smooth product launches are simpler than you think

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Committee on Accounting for Socioeconomic Status in Medicare Payment Programs; Board on Population Health and Public Health Practice; Board on Health Care Services; Health and Medicine Division; National Academies of Sciences, Engineering, and Medicine. Systems Practices for the Care of Socially At-Risk Populations. Washington (DC): National Academies Press (US); 2016 Apr 7.

Systems Practices for the Care of Socially At-Risk Populations.

• Hardcopy Version at National Academies Press

Appendix A Example Implementation Strategies and Case Studies

As described in Chapters 1 and 2 , the committee reviewed both the peer-reviewed and grey literature in order to identify strategies providers disproportionately serving socially at-risk populations have implemented to improve care and outcomes for their patients. As part of this effort, the committee reached out to organizations known to conduct research or represent providers disproportionately serving socially at-risk populations (Alliance of Community Health Plans, America's Essential Hospitals, America's Health Insurance Plans, and The Commonwealth Fund) who submitted 60 case studies. The committee also searched the published literature to identify additional examples. Based on a review of the case studies submitted, informed also by the literature and, in some cases, committee members' empirical research or professional experience delivering care to socially at-risk populations, the committee identified commonalities from which it concluded that six community-informed and patient-centered systems practices show promise for improving care for socially at-risk populations.

This appendix includes a series of tables that provide a selection of implementation strategies and case studies in which these strategies were identified for each of the six systems practices. Table A-1 includes examples regarding a commitment to health equity, Table A-2 includes examples of data and measurement strategies, Table A-3 has examples of components of comprehensive needs assessments, Table A-4 provides examples of collaborative partnerships, Table A-5 offers strategies for providing care continuity, and Table A-6 lists examples of engaging patients in their care. These tables aim to illustrate the range and types of activities that individual health care providers have implemented to apply each of the six systems practices. This appendix should therefore be considered a series of illustrative examples rather than a comprehensive and exhaustive list of organizations and practical strategies identified in the published and grey literature.

Commitment to Health Equity: Example Implementation Strategies and Case Studies.

Data and Measurement: Example Implementation Strategies and Case Studies.

Comprehensive Needs Assessment: Example Implementation Strategies and Case Studies.

Collaborative Partnerships: Example Implementation Strategies and Case Studies.

Care Continuity: Example Implementation Strategies and Case Studies.

Engaging Patients in Their Care: Example Implementation Strategies and Case Studies.

• ACHP (Alliance of Community Health Plans). Fallon Health Navicare and Summit Eldercare programs. n.d.-a. [December 14, 2015]. http://www ​.achp.org/wp-content ​/uploads/report-Innovation ​_ElderCare_FALLON.pdf .
• ACHP. Geisinger Health Plan's medically complex medical home program. n.d.-b. [December 14, 2015]. http://www ​.achp.org/wp-content ​/uploads/report-Innovation ​_ElderCare_GEISINGER.pdf .
• ACHP. Kaiser Permanente's PATHWAAY program. n.d.-c. [December 14, 2015]. http://www ​.achp.org/wpcontent ​/uploads/report-Innovation ​_ElderCare5.8.15_KP.pdf .
• ACHP. Priority Health's tandem365 program. n.d.-d. [December 14, 2015]. http://www ​.achp.org/wp-content ​/uploads/reportInnovation ​_ElderCare_PRIORITY.pdf .
• America's Essential Hospitals. Community-focused steps to boosting nutrition, wellness. 2014. [March 18, 2016]. http: ​//essentialhospitals ​.org/quality/community-focused-steps-to-boosting-nutrition-wellness .
• America's Essential Hospitals. Improving care transitions for socially, medically complex patients. 2015. [March 18, 2016]. http: ​//essentialhospitals ​.org/quality/improving-care-transitions-for-socially-medically-complexpatients .
• Cebul RD, Love TE, Einstadter D, Petrulis AS, Corlett JR. Metrohealth care plus: Effects of a prepared safety net on quality of care in a Medicaid expansion population. Health Affairs. 2015; 34 (7):1121–1130. [ PubMed : 26153306 ]
• Corrigan JM, Fisher ES. Accountable health communities: Insights from state health reform initiatives. 2014. [March 7, 2016]. http://tdi ​.dartmouth ​.edu/images/uploads/AccountHealthCommWhPaperFinal.pdf .
• Felland LE, Lechner AE, Sommers A. Improving access to specialty care for Medicaid patients: Policy issues and options. 2013. [March 8, 2016]. http://www ​.commonwealthfund ​.org/~/media/files ​/publications/fundreport ​/2013/jun/1691 ​_felland_improving ​_access_specialty_care_medicaid_v2.pdf .
• Foubister V. Case study: Louisiana's poor rankings make improving birth outcomes a state imperative. 2013. [February 4, 2016]. http://www ​.commonwealthfund ​.org/publications ​/newsletters/qualitymatters ​/2013/february-march ​/case-study .
• Hostetter M, Klein S. In focus: Innovating care delivery in the safety net. 2014. [February 2, 2016]. http://www ​.commonwealthfund ​.org/publications ​/newsletters/quality-matters ​/2014/december2014-january-2015/in-focus .
• Hostetter M, Klein S. In focus: Segmenting populations totailor services, improve care. 2015. [March 8, 2016]. http://www ​.commonwealthfund ​.org/publications ​/newsletters/quality-matters ​/2015/june/in-focus .
• Itzkowitz SH, Winawer SJ, Krauskopf M, Carlesimo M, Schnoll-Sussman FH, Huang K, Weber TK, Jandorf L. New York Citywide Colon Cancer Control Coalition: A public health effort to increase colon cancer screening and address health disparities. Cancer. 2016; 122 (2):269–277. [ PMC free article : PMC4939480 ] [ PubMed : 26595055 ]
• Klein S. Colorado Coalition for the Homeless: A model of supportive housing. 2014. [February 2, 2016]. http://www ​.commonwealthfund ​.org/publications ​/newsletters/quality-matters ​/2014/octobernovember ​/case-study .
• Klein S, McCarthy D. North Carolina's ABCD program: Using community care networks to improve the delivery of childhood developmental screeing and referral to early intervention services. 2009. [March 8, 2016]. http://www ​.commonwealthfund ​.org/publications ​/issue-briefs/2009 ​/aug/north-carolinasabcd-program-using-community-care-networks-to-improve-the-delivery . [ PubMed : 19697485 ]
• Klein S, McCarthy D. Genesys HealthWorks: Pursuing the triple aim through a primary care-based delivery system, integrated self-management support, and community partnerships. 2010. [February 2, 2016]. http://www ​.commonwealthfund ​.org/publications ​/case-studies/2010 ​/jul/genesys-healthworks .
• Klein S, McCarthy D, Cohen A. Health Share of Oregon: A community-oriented approach to accountable care for Medicaid beneficiaries. 2014a. [February 2, 2016]. http://www ​.commonwealthfund ​.org/publications ​/case-studies/2014 ​/oct/health-share-oregon-acocase-study .
• Klein S, McCarthy D, Cohen A. Tuscon and southern Arizona: A desert region pursuing better health and health system performance. 2014b. [March 8, 2016]. http://www ​.commonwealthfund ​.org/publications ​/case-studies/2014 ​/apr/tucson-and-southernarizona-a-desert-region-pursuing-better-health-and-health-system-performance .
• Lovelace J. Integrating health care and supported housing to improve the health and well-being of the homeless: A population health case study. 2016. [March 29, 2016]. http://nam ​.edu/integrating-health-care-andsupported-housing-to-improve-the-health-and-well-being-of-the-homeless-a-population-healthcase-report/
• McCarthy D, Chase D. Montefiore medical center: Integrated care delivery for vulnerable populations. 2010. [March 8, 2016]. http://www ​.commonwealthfund ​.org/publications ​/case-studies/2010 ​/oct/montefioremedical-center .
• McCarthy D, Mueller K. The New York City health and hospitals corporation: Transforming a public safety net delivery system to achieve higher performance. 2008. [February 4, 2016]. http://www ​.commonwealthfund ​.org/~/media/files ​/publications/fund-report ​/2008/oct/the-newyork-city-health-and-hospitals-corporation--transforming-a-public-safety-net-deliverysystem ​/mccarthy ​_nychlthospitalscorpcasestudy ​_1154-pdf.pdf .
• McCarthy D, Beck C, Nuzum R, Gauthier A. Denver Health: A high-performance public health care system. 2007. [March 18, 2016]. http://www ​.commonwealthfund ​.org/publications ​/fundreports/2007 ​/jul/denver-health--a-high-performance-public-health-care-system .
• McCarthy D, Klein S, Cohen A. Opportunity for regional improvement: Three case studies of local health system performance. 2014. [March 8, 2016]. http://www ​.commonwealthfund ​.org/~/media/files ​/publications/casestudy ​/2014/apr/1737 ​_mccarthy_regional ​_case_study_synthesis_v2.pdf .
• Meyers K. Beyond equal care: How health systems can impact racial and ethnic health disparities. 2008. [February 2, 2016]. http://share ​.kaiserpermanente ​.org/media_assets ​/pdf/communitybenefit ​/assets/pdf/our_work ​/global/BeyondEqualCare2001_08 ​.pdf . [ PMC free article : PMC3042345 ] [ PubMed : 21369518 ]
• Murray M, Bodenheimer T, Rittenhouse D, Grumbach K. Improving timely access to primary care: Case studies of the advanced access model. Journal of the American Medical Association. 2003; 289 (8):1042–1046. [ PubMed : 12597761 ]
• Sandberg SF, Erikson C, Owen R, Vickery KD, Shimotsu ST, Linzer M, Garrett NA, Johnsrud KA, Soderlund DM, DeCubellis J. Hennepin health: A safety-net accountable care organization for the expanded Medicaid population. Health Affairs. 2014; 33 (11):1975–1984. [ PubMed : 25367993 ]
• Silow-Carroll S, Rodin D. Forging community partnerships to improve health care: The experience of four Medicaid managed care organizations. 2013. [February 2, 2016]. http://www ​.commonwealthfund ​.org/publications ​/issue-briefs/2013 ​/apr/forging-communitypartnerships-to-improve-care . [ PubMed : 23634464 ]
• Cite this Page Committee on Accounting for Socioeconomic Status in Medicare Payment Programs; Board on Population Health and Public Health Practice; Board on Health Care Services; Health and Medicine Division; National Academies of Sciences, Engineering, and Medicine. Systems Practices for the Care of Socially At-Risk Populations. Washington (DC): National Academies Press (US); 2016 Apr 7. Appendix A, Example Implementation Strategies and Case Studies.
• PDF version of this title (1.7M)

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Example Implementation Strategies and Case Studies - Systems Practices for the C... Example Implementation Strategies and Case Studies - Systems Practices for the Care of Socially At-Risk Populations

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

UC San Diego

Restorative Justice in Higher Education: A Case Study of Program Implementation and Sustainability

• LaCroix, Luke Leo
• Advisor(s): Huie Hofstetter, Carolyn

ABSTRACT OF THE DISSERTATION: Restorative Justice in Higher Education:

A Case Study of Program Implementation and Sustainability by Luke Leo LaCroix, Doctor of Education in Educational Leadership, University of California, San Diego, 2018, California State University, San Marcos, 2018 and Carolyn Huie Hofstetter, Chair

Restorative justice is an emerging topic related to college student behavioral issues and offers a personal behavior alternative strategy. It is used predominantly to address crime, misconduct, and injustices in criminal justice, K-12 education, and higher education arenas. Restorative justice addresses harm through dialogue with goals of repairing harm, rebuilding trust, and repairing relationships through a mutual decision-making process in order to determine outcomes with responsibilities for all participants. The literature demonstrates how restorative justice complimented student conduct processes, developed empathy, had a positive impact on school and campus culture, and the advancement of cultural and diverse competencies, and incurred great satisfaction by the participant.

Literature further states that restorative justice is supported by many professional groups in the Roman Catholic higher education setting as a personal behavior alternative strategy and reflects the spirit of Catholic Social Teachings.

This study examined how a Roman Catholic university implemented and sustained a restorative justice program at one site in southern California. Data included interviews and focus groups with leaders and facilitators working actively within the program. Emergent themes from the qualitative data included: institutional need, shared belief system, university identity, Catholic Social Teachings, proactive use of restorative justice and restorative practices, reactive use of restorative justice and restorative practices, systematic training, collaboration, institutional support, institutionalizing the program, storytelling, reframing current work, frequent use of restorative justice practices impact of program sustainability on campus, impact of cultural experience, empathy, conflict management, local and national recognition, and student voice.

Key words: restorative justice, restorative practices, Roman Catholic, mission, empathy, conflict management, storytelling, collaboration, college, university, higher education

Enter the password to open this PDF file:

Lessons from the field: case studies in global implementation science

Collection Organisers: Blythe Beecroft, Fogarty International Center, USA ; Gila Neta, National Cancer Institute, USA ; Rohit Ramaswamy, Cincinnati Children's Hospital Medical Center, USA ; and Rachel Sturke,  Fogarty International Center, USA

Fogarty’s Center for Global Health Studies (CGHS), in collaboration with the Cincinnati Children’s Hospital Medical Center and the National Cancer Institute (NCI), has developed a collection of case studies documenting examples of rigorous implementation research in low- and middle-income countries (LMICs).  For an in-depth introduction to the collection, please refer to the commentary: The “case” for case studies: why we need high-quality examples of global implementation research .

Documented examples of rigorous implementation research in global contexts can be a valuable resource and help build research capacity.  The overall collection is intended to highlight the value of case studies as a source of “thick” (i.e., context-rich) description and a teaching tool for global implementation researchers. It addresses the independent merit of case studies as an evaluation approach for disseminating high quality research in a format that is useful to a broad range of stakeholders. Additionally, case studies can strengthen research by demonstrating a variety of research designs to answer complex questions and generating generalizable knowledge about implementation. 

These research articles, published in this collection in Implementation Science Communications , provide practical examples of how to select, adapt, and apply implementation science models, theories, or frameworks to global settings, develop and test implementation strategies, and evaluate implementation processes and outcomes. 

All case analyses in the collection have undergone the journal’s standard peer-review process and span various disease areas across diverse international contexts. 

This collection would not be possible without the CGHS Steering Committee and the help of Bryan Weiner, Christopher Gordon, Eche Ezeanolue, Ejemai Eboreime, and Suzy Pollard.

Implementation of new technologies designed to improve cervical cancer screening and completion of care in low-resource settings: a case study from the Proyecto Precancer

This case study details the experience of the Proyecto Precancer in applying the Integrative Systems Praxis for Implementation Research (INSPIRE) methodology to guide the co-development, planning, implementati...

• View Full Text

mHealth to improve implementation of TB contact investigation: a case study from Uganda

Implementation science offers a systematic approach to adapting innovations and delivery strategies to new contexts but has yet to be widely applied in low- and middle-income countries. The Fogarty Center for ...

Tracking adaptation strategies of an HIV prevention intervention among youth in Nigeria: a theoretically informed case study analysis of the 4 Youth by Youth Project

Although many behavioral interventions are adapted, little is known about the reasons for adaptations and the process and outcomes influencing adaptations. To address this gap, we explored the adaptations made...

Assessing the implementation fidelity of HPV self-collection offered by community health workers during home visits (the EMA strategy): a case study in a low-middle-resource setting in Argentina

In Argentina, HPV  self-collection offered by community health workers was demonstrated to be effective to improve cervical cancer screening uptake. Based on these findings, the EMA strategy was scaled up in n...

Adapting a tobacco cessation treatment intervention and implementation strategies to enhance implementation effectiveness and clinical outcomes in the context of HIV care in Vietnam: a case study

Smoking rates remain high in Vietnam, particularly among people living with HIV/AIDS (PLWH), but tobacco cessation services are not available in outpatient HIV clinics (OPCs). The research team is conducting a...

Technology-assisted task-sharing to bridge the treatment gap for childhood developmental disorders in rural Pakistan: an implementation science case study

As in many low-income countries, the treatment gap for developmental disorders in Pakistan is nearly 100%. The World Health Organization (WHO) has developed the mental Health Gap Intervention guide (mhGAP-IG) ...

Alternate delivery platforms and implementation models for bringing evidence-based behavioral interventions to scale for youth facing adversity: a case study in West Africa

Youth Functioning and Organizational Success for West African Regional Development (Youth FORWARD) was launched as an implementation science collaboration focused on scaling out evidence-based mental health in...

The “case” for case studies: why we need high-quality examples of global implementation research

Rigorous and systematic documented examples of implementation research in global contexts can be a valuable resource and help build research capacity. In the context of low- and middle-income countries (LMICs)...

• Skip to content
• Skip to search
• Staff portal (Inside the department)
• Student portal
• Key links for students

Other users

• Forgot password

Notifications

{{item.title}}, my essentials, ask for help, contact edconnect, directory a to z, how to guides.

• Centre for Education Statistics and Evaluation

Curriculum implementation case studies

These case studies were originally published 24 May 2023.

• Berrigan Public School (PDF 3456 KB)
• Biraban Public School (PDF 2222 KB)
• Fairfield West Public School (PDF 5752 KB)
• Parklea Public School (PDF 4536 KB)
• Wairoa School (PDF 3798 KB)
• Walcha Central School (PDF 4412 KB)

In 2022, 396 schools participated in a pilot project to trial the implementation of the K-2 section of the new K-10 English and Mathematics syllabuses. These early adopter schools implemented the new English and Mathematics syllabuses with their Year 1 cohorts, testing supports such as microlearning modules, scope and sequences, and sample units.

The Centre for Education Statistics and Evaluation (CESE) selected six of these early adopter schools to identify strategies they reported as having worked well in the first year of implementing the new syllabuses with their Year 1 cohorts. The case study schools represent the diverse settings of NSW public schools.

Across the six case studies, school leaders and teachers share strategies that helped them to effectively implement the new curriculum in their school’s context. These strategies are grouped into four themes:

using effective leadership and change management approaches

creating a school culture receptive to the new curriculum

adapting school structures and processes

developing quality classroom-based practices.

How to use this resource

About this resource.

In 2022, 396 schools across NSW participated in a pilot project to enhance the NSW Department of Education’s development and delivery of comprehensive support for implementation of the new English and mathematics K–10 syllabuses.

These early adopter schools implemented the new English and mathematics syllabuses with their Year 1 cohorts, testing supports such as microlearning modules, scope and sequences, and sample units. Implementation support developed and refined in 2022, as a result of feedback from participating schools, is available to all schools ahead of mandatory implementation of the new syllabuses for Kindergarten to Year 2 from 2023.

This series of case studies by the Centre for Education Statistics and Evaluation (CESE) are about how early adopter schools have begun implementing new curriculum. Each case study shares how the focus school has engaged with the new syllabuses and explores key themes in how they approached curriculum implementation responsive to their context. The evidence base for the case studies can be found in the department’s curriculum implementation research toolkit .

When and how to use

School leaders and teachers can read, consider, discuss and implement strategies highlighted in the case study and its accompanying discussion guide as part of school‑developed High Impact Professional Learning (HIPL).

The appropriate time to use this resource may differ for each school, leader and teacher.

School leaders can:

• unpack the case study and curriculum resources as part of whole-school professional development and/or stage or grade team meetings
• access the phases of curriculum implementation and role-specific resources to support reflection and planning for effective implementation of curriculum
• access their school’s data to lead discussions with staff about areas for improvement at a classroom and whole-school level.

Teachers can:

• read the case study and curriculum resources and reflect on current practice
• consider which practices could be implemented in the classroom to lift student achievement, and discuss their thoughts with colleagues
• reflect on the impact of implementation.

Email feedback about this resource to [email protected]. You can also subscribe to the CESE newsletter and connect with us on Yammer .

Alignment to system priorities and/or needs: NSW Curriculum Reform, NSW Department of Education Literacy and numeracy priorities, School Excellence in Action, Mathematics Strategy 2025

Alignment to School Excellence Framework: Learning domain – curriculum, assessment; Teaching domain – professional standards, effective classroom practice, data skills and use; Leading domain – educational leadership, school planning, implementation and reporting

Alignment with other existing frameworks: High Impact Professional Learning , What works best , Curriculum planning and programming, assessing and reporting to parents K–12 , Literacy K–12 , Numeracy K–12 , Aboriginal Education , Inclusive Education for students with disability , High Potential and Gifted Education , Multicultural Education , Controversial Issues in Schools

Reviewed by: Curriculum and Reform; Educational Standards; Teaching Quality and Impact; Inclusion and Wellbeing

Created/last updated: Originally published 23 May 2023.

To be reviewed: CESE publications are prepared through a rigorous process. Resources are reviewed periodically as part of an ongoing evaluation plan.

Related resources

• Curriculum implementation discussion guide

Business Unit:

NetxInvestor  arrow_forward

Netx360+  arrow_forward

Wove Platform arrow_forward

Wove Connect arrow_forward

Case Study: Management Training Program

Registered investment advisor, this pershing client is an ria looking to create efficiencies through technology.

After the event, the Pershing Business Consultant set up monthly “Office Hours” meetings with the group of 8 to discuss lessons learned from the field, putting the content into practice, and diving deeper into managerial skills.

A large wealth management firm with multiple locations had over the last year promoted several staff into people management/team management roles and none of these individuals had any prior experience or training on managing people. The organization, whilst large, did not have the resources to train this group internally.

Pershing’s Business Consultant proposed a one day “boot camp” training program to introduce the basics of people and team management to the group of 8 managers. The content consisted of:

• Managing Yourself
• Cadence of Managerial Duties
• Managing High Performers
• Managing Under Performers
• How to Set Your Team’s Vision
• Setting Success Metrics

The content was delivered in lecture format, self-reflection, group reflection, role plays and a panel discussion.

Ask how we can help transform your business

Additional resources, choose your region or location.

Implementing a Preventive Maintenance Strategy: Case Study

by Drew MacRae, Training & Solutions Manager, Reliability

Preventive Maintenance (PM) is an important part of any facilities management strategy. The goal of a successful preventive maintenance program is to establish consistent practices to improve the operation and safety of the equipment at your site. It also aims to reduce equipment breakdowns and the number of major corrective maintenance tasks.

Reactive vs. Preventive Maintenance

Poor maintenance strategies can significantly reduce an organization’s productive capacity. Many companies use a reactive maintenance strategy where repairs to an asset are made once a failure occurs. This strategy has significant cost implications due to critical operational outages and rush orders for parts and labour.

Instead, effective PM programs are designed to prevent failures before they ever have the chance to occur by proactively performing maintenance, repairs, and replacements. Preventive Maintenance tasks may include inspections, lubrication, oil changes, adjustments, repairs, replacing parts, or even regularly scheduled partial or complete overhauls.

Implementing a PM program often requires a shift in the organization’s overall maintenance strategy. The technicians doing the work need to buy into the program as it allows them to work alone without close supervision and to have more ownership of the equipment. It is also important to regularly audit the program: adding new equipment and removing obsolete equipment as required.

Case Study: Implementing a PM Strategy

The challenge:.

A large oil and gas equipment manufacturing facility asked Fluid Life to help them with their maintenance programs. When we arrived on site we found out that there was no systematic maintenance program in place; they weren’t using a Computerized Maintenance Management System (CMMS); and they weren’t planning or scheduling maintenance tasks. Their strategy? When equipment broke down, there was a backup asset. If the backup broke down, they would source another backup via rental or new purchase. Generally technicians would tend to wait until something went wrong before acting.

The Solution:

Our Reliability Specialist completed a full review of their current maintenance strategies and programs and put together a plan to implement preventive maintenance across the organization. To develop the plan we assessed every asset on the site, completed a criticality analysis, and determined the correct PM for each asset. Bringing in PM’s was opposite to what the team was used to and there was some push-back. We gained buy-in by including maintenance staff in every step as to what needed to be done and why.

We also implemented our myLab online program management portal to use as a CMMS. This allowed the customer to plan and schedule all maintenance tasks. Furthermore the customer had step by step instructions on how the technicians are to complete the task and provide comments to schedule maintenance for equipment that required corrective maintenance.

The Result:

The customer was pleased with the new maintenance program outlined and have been using this for over a year now. They can now more easily plan and schedule when preventative maintenance tasks need to be completed and ensure that their equipment is properly maintained. They’ve also reduced the high costs of corrective maintenance.

Interested in learning more? Contact our team of reliability specialists and find out how we can help you develop a preventative maintenance program, or optimize your current PM program.

• News & Analysis on Food & Beverage Development & Technology

Food-as-medicine case study: Ahold Delhaize USA shares multi-discipline approach for successful programs

28-May-2024 - Last updated on 28-May-2024 at 14:45 GMT

• Email to a friend

Ahold Delhaize USA operates several grocery chains on the East Coast, including Giant Food, Hannaford, Food Lion and Stop & Shop. Ahold Delhaize USA serves 30 million customers a week in both rural and urban settings, said Steven Jennings, stakeholder relations and brand lead, health and sustainability and founder for the retailer.

Ahold Delhaize USA ditches the food as medicine naming convention, uses ‘Foodcare’ instead 

When Ahold Delhaize USA developed its food-as-medicine discipline, executive leadership and stakeholders worried how consumers might interpret the phrase "food as medicine," Jennings admitted. 

Instead, the retailer decided to use its own terminology and called the overall strategy Foodcare, he said. The debate on whether or not to use the food as medicine phrase was a reoccurring point of discussion in various event sessions, including the opening panel ​.  

“Food as medicine sounds a little scary for some of our internal partners or stakeholders. ... We are there to hold their hands on this journey, and Foodcare is about inspiring and empowering consumer choice on their healthier living journeys. It is not about shaming. It is not about pointing to this or that. It is about meeting our customers and associates where they are on their specific healthcare living journey," he said.

Echoing the sentiment,   ​Mandy Katz, director of healthy living at Giant Food, explained that Foodcare helped her team communicate with executive leadership about the goals of various programs and receive executive approval for projects.   

Ahold Delhaize USA’s Foodcare initiative focuses on three core pillars — guide, curate and nourish   ​— he added. 

Giant Food food-as-medicine wins: Incentive buying, Produce Rx programs ​

Katz shared "two really big wins” for Ahold’s Foodcare strategy and the lessons her team learned in the process.

Giant Food’s customer incentive program rewards consumers for purchasing healthier food was one success, Katz explained. Additionally, Giant Food’s first Produce Rx Program ​ — where Medicaid recipients received a $20 coupon to purchase fresh fruits and vegetables at Giant stores — helped the organization understand how best to develop a food-as-medicine program moving forward, she added.

“We learned how those little vouchers that we started with do not work in our stores. We learned about gift cards and how gift cards are lovely ... but you cannot filter them, and you do not know if the partners who are getting and buying those cards are actually able to distribute them, and the people receiving them are using [the gift cards] for the things they want, or are they buying other essentials like toothpaste and toilet paper — which is just as important — but it does not go towards the programming," she said.

Using Guiding Stars to make shopping for healthier foods easier 

Ahold Delhaize USA partnered with nutrient-density scoring company Guiding Stars Licensing Company to provide a visual cue in-store for consumers searching shelves for healthier foods, said Elizabeth Marble Caton, brand and nutrition manager at Guiding Stars Licensing Company. 

Guiding Stars takes nutritional information for foods and beverages sold at Giant Food stores and assigns the product a 1-3 star ranking for good, better and best to inform consumers on what is the most nutritious, Carton explained. The ranking is depicted by an apple icon with the corresponding number and is displayed next to a price tag in Giant Food stores.  

Guiding Stars not only helps consumers find better products, but it helps inform how Ahold Delhaize USA markets healthier and more nutritious foods and beverages, Katz noted. 

"Guiding Stars is our love language that we speak constantly to our merchandising team. ... [Our healthy merchandising manager] is constantly working with Elizabeth, figuring out why products may or may not earn stars, but she is also helping us to look at the way we market health in-store,” she added.   

Partnering with IT for successful food-as-medicine programs

In developing its Foodcare programs, Ahold Delhaize USA addressed several technology limitations, which impede the implementation of several programs, the panelists shared. 

When Ahold Delhaize USA started accepting a health benefits card from a third-party vendor, the retailer’s payment system was unable to process the card transaction, which frustrated consumers, Jennings explained. 

The Ahold Delhaize USA Foodcare team requested and received “a non-budgeted IT upgrade to the tune of ... several million dollars" to make the benefit cards work with the retailer’s payment system, he said.

The Foodcare team used this experience to shore up its communication and collaboration with Ahold Delhaize USA’s IT team and technology partners, including electronic-voucher company iQPay, to ensure this issue would not happen again, Jennings said. 

“We cannot invest that money on the slim margins that we operate on for every single card that comes along the way. So, this is when working with our IT partners, we created internal teams [to do] discovery around what are all the cards of market [and] what are the most common systems."

Related news

Related products

Gain an edge with our active nutrition expertise

Content provided by ADM | 25-Apr-2024 | White Paper

The intersection of the rising plant-based trend and increasing awareness of gut health has opened new opportunities for market success.

Replacement Isn't the Future. Variety Is.

Content provided by ADM | 22-Mar-2024 | White Paper

Successfully navigating the intersection of food and technology can help your business meet evolving consumer demands.

Some home truths about real prebiotic dietary fibre

Content provided by BENEO | 22-Mar-2024 | Product Presentation

Confused about prebiotics? You’re not the only one! Food developers wanting to work with prebiotic dietary fibre are faced with an abundance of products...

Consumer Attitudes on Ultra-Processed Foods Revealed

Content provided by Ayana Bio | 12-Jan-2024 | White Paper

Ayana Bio conducted the Ultra-Processed Food (UPF) Pulse survey, offering insight into consumers’ willingness to consume UPFs, as well as the variables...

Related suppliers

• Blue Diamond Global Ingredients
• IMBAREX Natural Colors
• Rethink Events Ltd
• How color impacts the food & beverage experience Givaudan | Download Technical / White Paper
• Tales of Triumph from the Factory Floor, Vol. 1 QAD Redzone | Download Case Study
• Gain an edge with our active nutrition expertise ADM | Download Technical / White Paper
• Learn more: Future Food-Tech Alternative Proteins Rethink Events Ltd | Download Technical / White Paper
• Sustainably sourced. Naturally versatile. Covation Bio™ PDO | Download Insight Guide
• Replacement Isn't the Future. Variety Is. ADM | Download Technical / White Paper

Upcoming editorial webinars

• 24 Sep 2024 Tue Webinar Decarbonising Supply Chains at the Source
• 25 Sep 2024 Wed Webinar Eco-friendly Production from Farm to Fork
• 26 Sep 2024 Thu Webinar New Tech and Novel Ingredients for a Sustainable Future

On-demand webinars

• Plant-based 3.0: Emerging from the trough of disillusionment Webinar
• Unpacking the Impact: What State Color and Food Additive Bans Mean for Your Products FoodChain ID
• The easy path to “dry” cleaning & sanitization
• From Labeling to Leading: A Cost-Effective Approach for Empowering Innovation FoodChain ID
• Real-time Brix monitoring - the secret ingredient your process needs for quality and efficiency
• Sweet Solutions: Exploring the Future of Sugar Reduction

Promotional Features

FoodNavigator-USA

• Advertise with us
• Apply to reuse our content
• Press Releases – Guidelines
• Contact the Editor
• Report a technical problem
• Whitelist our newsletters
• Why Register
• Editorial Calendar
• Event Calendar

Evolution of environmental chemistry study program curricula in tertiary education: a case study and general implications

• Environmental Chemistry for A Pollution Free Society
• Open access
• Published: 01 June 2024

Cite this article

You have full access to this open access article

• Peter Šebej   ORCID: orcid.org/0000-0003-0317-0630 1 &
• Jakub Urík 1  

Societal and scientific progress has led to the formation of new study programs, often with multidisciplinary curricula. Guarantors and teachers of such programs must be prepared to quickly adapt to the needs and demands of students, society, the job market, and the commercial sphere since many issues start surfacing during the first years of the programs’ life. Here we share our experience with such a process in the study program “Environment and Health” taught since 2019 at Masaryk University in Brno, Czech Republic. Feedback from students and alumni allows for improvement of the curriculum and organization of the program. We show feedback loops from three perspectives: Feedback from immediate short-term experience can be incorporated within a year, medium-term feedback loops can manifest after several years, and long-term ones even decades. While current students usually perceive only short- or medium-term issues, the philosophy and structure of the program must be built by predicting societal and commercial needs in the following decades. Such long-term aspects are often counterintuitive to students’ vision, but still have to be considered for the program to remain attractive to new applicants. Balancing the original vision, preparing and applying changes, and dealing with feedback on all levels are key managerial challenges of successful study programs.

Graphical Abstract

Avoid common mistakes on your manuscript.

Introduction

Tertiary education is (typically) organized in large institutions (i.e., universities, colleges, professional and other schools, and here we will term all of them under a generic university ) that are also actively engaged in research and other societal activities such as outreach, applied and/or for-profit development, consulting, and policy development.

At contemporary institutions, the students usually choose one study program (SP) that offers a multitude of courses, constituting a curriculum with pre-defined learning and training outcomes and an alumnus profile. The particular courses follow a course syllabus. An accreditation body typically assures and controls the quality of education and decides on the accreditation of SPs and/or institutions. Advances in knowledge, scientific discoveries, and societal changes have usually been quickly transferred into tertiary education, leading to pressure on changes in curricula. Quantitative and semi-quantitative comparison of SPs across European universities is simple using The European Credit Transfer and Accumulation System (ECTS)—a tool of the European Higher Education Area proposed by the Bologna declaration in 1999 (European Commission and Directorate-General for Education 2015 ).

Creation and design of new study programs

In many fields, SPs and their curricula seem to be relatively stable and do not change quickly. However, even in such fields, we can historically trace changes and creations of new SPs, often by branching existing ones, in the same way as we can trace the “creation” of new scientific fields. This process often follows the lifecycle of the scientific paradigm shift, as described by Kuhn (Kuhn and Hacking 2012 ). A good example could be sprung out in the chemical disciplines. Organic chemistry stemmed from (general) chemistry. Biochemistry stemmed from organic chemistry. Others resulted from interactions of two or more fields, e.g., molecular biology arose where chemistry was necessary to understand biological processes. Environmental chemistry arose where physical, organic, and other chemical disciplines were necessary to understand processes in nature, and analytical science became a tool. In both these scenarios leading to new SPs, their creation and curation require strategic planning, with considerations of the skills and knowledge of future alumni and their participation in the job market in the long term. For programs arising at interfaces of fields, building and balancing the interdisciplinarity of SPs are also very important.

To simplify the dynamics of SPs, we could divide the relevant processes into (1) the formation of SPs; and (2) changes, evolution, and interactions during the program life cycle. These processes might span decades or more. At European universities, more specialized SPs (including at the bachelor level) are typical; however, at US universities, rather general SPs prevail. Discussions about the scope, depth, syllabi, and details of a new SP are especially crucial for the more specialized SPs, as they possess a higher risk of increased dropout ratio, fueled by students who realize the particular specialization does not suit them only during the studies.

Why do we need feedback?

To keep potential students interested, actual students motivated, and alumni relevant for the job market, the curation of the SP curriculum is important. Inputs for curriculum management could come from feedback from all these groups, as well as from faculties, university management, and other stakeholders too (Benneworth and Jongbloed 2010 ; Jain et al. 2022 ). This idea is parallel to the fact that the success of for-profit companies depends on having working closed feedback loops, and such a strategy was also suggested to be implemented in governance (Whittle 2016 ). However, students’ feedback is primarily discussed within the context of a particular course or study unit (Malecka et al. 2022 ), but not so much in the context of SPs—higher hierarchy study units.

In this paper, we would like to shed light particularly on the evolution of SPs and feedback loops involved in this process and comment on the evolution of curricula of the study program Environment and Health (EH) at our institution as a case study and an example that allows us to suggest some generalizations. Our point of view is thus focused on programs in the sciences, especially those related to life, health, and environmental sciences.

Emerging study programs and environmental chemistry

Many tertiary education SPs have names that give a very clear idea about the curriculum content to the general audience, e.g., biology, chemistry, and mathematics, and many others, especially at technical, trade, or other specialized institutions, also imply a clear connection to a particular job, e.g., chemical engineering or medical genetics. With the exception of programs with a connection to a professional career pathway, these are almost always representations of traditional and established fields. On the other hand, many of the emerging or multidisciplinary curricula, despite being created with very good reasoning, job-market research, and SWOT analysis, could run into issues of communication, misleading names, general misunderstandings, or even miss the potential auditoria (prospective students, employees, etc.).

The origins of SPs focused on environmental chemistry trace back to research on the observable effects of pollution on humans (e.g., “The great London smog”) (Stone 2002 ) or ecosystems (e.g., of DDT on populations of eagles) (Carson 1962 ). These have often been very closely related to analytical and physical chemistry but have been expanded to other subdomains of chemistry. In a recent study, it was shown that there are inconsistencies, and significantly variable shares of time/credits are now allocated to courses covering various chemical disciplines (Lammel et al. 2014 ). At about the same time, the formation of SPs in toxicology, ecotoxicology, and other fields relevant to the environment was advancing. Over time, the relationship between chemicals, environment, and health became the key concept in environmental and related fields, and the concept of the exposome with all its roles is now discussed in detail (Gao 2021 ; Barouki et al. 2022 ; Price et al. 2022 ). The concept of One Health similarly interconnects human, animal, and environmental health (Schwabe 1984 ; Zinsstag et al. 2011 ; Destoumieux-Garzón et al. 2018 ; Gao 2021 ). The emergence of such interdisciplinary concepts, research topics, approaches, paradigms, and changes in the philosophy of particular fields is often followed by the development of the SPs. In environmental sciences, this has led to the appearance of integrative programs covering a wide range of topics in environmental sciences and health, including the one at Masaryk University.

Study program Environment and Health

Our SP Environment and Health (EH)—bachelor ( a ) and following master ( b )—were created with the philosophy of education in the fields related and important for understanding environment and health, which translated into key requirements of:

Solid chemical and biological background to build upon (mainly a )

Orientation in environmental and (eco)toxicological issues (basics in a , in-depth in b )

Interdisciplinary understanding and ability to interconnect knowledge (mainly b )

Understanding and hands-on experience in a wide array of laboratory and field skills (both a and b )

Presentation, teamwork, and other soft (transferable) skills (both a and b )

These programs started at Masaryk University in 2019 at the bachelor (3 years) and master (2 years) levels (see Supporting Information for details on organization of the tertiary education in the Czech Republic) and have been open for applications every year since. Typically, each year there are about 100–120 applicants and about 30 students starting the first year of bachelor SP.

Feedback loops

To learn more about students’ views on the programs and expectations and to get inputs for data-based decisions on program management, we are continuously investigating the motivation of students, their understanding of the SP, their views, and opinions, by questionnaires, interviews, offering counseling by designated faculty members from the Department, and specialized counselors or ombudspersons at the level of school or university. Additional feedback is facilitated by an SP board (see Supporting Information for more details). Across the entire university, students evaluate courses and teachers using an anonymous questionnaire at the end of each term. Altogether, these represent a wealth of pathways for getting feedback from students on various aspects of courses, curricula, etc.

We divided the feedback into three roughly defined feedback loops: (a) short-term, (b) medium-term, and (c) long-term (Fig.  1 ). The division is based on the feedback’s nature and content; potential implementation of changes, suggested responses, and actions; and the time necessary for implementation and observation of the effects. The simplest questions and suggestions are often regarding the quality of in-course teaching, course syllabi, course integration in the curriculum (incl., e.g., mandatory/voluntary course status, substantial changes in syllabus), and addition of a new course or a course withdrawal, which all could be gathered, processed, and analyzed very quickly. In many cases, the changes based on the feedback analysis could be implemented within the next academic year (i.e., typical reaction and implementation time is 1–2 years) and are considered here as short term. Medium feedback loops involve, e.g., job market situation, increase in or even newly appeared demand for particular skill sets, emerging new jobs or jobs ceasing to exist, and societal changes, particularly short term. These are sometimes expressed, e.g., by large funding schemes (such as EU’s Framework Programmes for Research and Technological Development, with Horizon Europe running now) (European Commission 2023 ). Other important inputs are the recently launched European Skills Agenda for Sustainable Competitiveness, Social Fairness and Resilience (European Commission 2020 ) and the already set global goals of sustainable development (United Nations 2015 ). Lastly, the long-term loops involve analysis and predictions of whole-career pathways, long-term society changes such as the role and function of tertiary education and philosophy of education. From this perspective, the whole concept of education looks very different, and this perspective is not the focus of this work.

Lifetime of the study programs and a simplified formal diagram of feedback loops of different timescales

During the 5 years of existence of our SP Environment and Health (both bachelor and master), while collecting feedback in all the abovementioned ways, we often heard students sharing the same view, paraphrased: “I never thought, that there will be so much chemistry in the environmental sciences.” On the contrary, professionals in research and development acknowledge the relevance of the concept of “Chemistry as a central science” (Balaban and Klein 2006 ; Brown et al. 2022 ) in environmental sciences. The understanding of the need for such a profound chemical background in the curriculum was also reported by our alumni. However, a share of students reported that they were interpreting the name of the SP as one focusing on environmentalism, not environmental chemistry, and on health outcomes of environmental issues, not mechanisms and risks. This was answered with several responses. The quick response was the introduction of a 2-h, full-term course (2 ETCS) on chemical calculations (with a dedicated instructor and small groups of students, not more than 20) and several seminars explaining the concepts of the SP, exposome, and overview of interrelationships of environment, health, and necessary basic fields and changes in communication of the program to potential students. The space devoted to inorganic chemistry was reported by students to be unnecessarily high, as the original concept contained courses evenly covering all basic chemical disciplines. We significantly cut it down and the freed space was reallocated to chemistry calculations—a technical transferable skill itself (all within the first year of study). The medium-term response was the introduction of two specializations—(a) environmental chemistry and toxicology and (b) environmental health, starting in 2023. These immediate simple responses, such as the introduction of the chemical calculations course, had a very quick effect of decreasing the dropout rate from the study (from ca. 40% before to ~ 20% after) and decreasing the rate of students leaving for another SP.

Students who advanced into later years of study and alumni generally provide positive feedback about practical courses (laboratory and field) in the curriculum. These are relatively evenly distributed throughout all years of EH SP in the bachelor level and first year of the master level, which is generally appreciated by students, as it helps keep their motivation high by offering a chance to acquire hands-on practical experience throughout the study. However, we observed students asking for courses more closely related to health and environment, as the first 2 years of the curriculum are mainly introductory to many basic biology and chemistry topics, including practical courses (see Supporting Information for details). To follow this demand and this motivation channel, a field course in the first year, as well as an expanded time for a general introductory seminar, was added to the curriculum. In the master’s SP, the situation is different; the practical and hands-on courses are already specialized, and a fair share of them is based on a problem-solving approach (Jansson et al. 2015 ) and is well accepted by students.

In line with the rapidly changing job market, emerging new skill sets and vanishing interest in others, and along a perspective of preparing alumni who will be pursuing their professional careers for about the next four decades, we put an emphasis on soft- and transferrable skills and bolster the will for and confidence in life-long learning as the key concept. To create space for their roles in the job market—both as a participant and as a creator—a series of mandatory seminars on transferrable skills (during the first 2 years of bachelor SP; 1 ETCS per term) was created and tailored to the anticipated needs of the EH SP. They cover areas such as (a) basic IT/software tools and knowledge, (b) basic scholarly skills, (c) scientific thinking, methods, and approaches to problems, and (d) presentation skills. On the basis of anonymous course feedback questionnaires and interviews with students, we found that they typically assess the courses and/or particular topics in the curricula through the lens of usefulness, especially in the short-term horizon (Fig.  2 ). We thought that this would lead, e.g., to understanding the usefulness of classes on skills such as primary and scientific literature search and writing. On the other hand, students often find the scientific thinking and research skills, shaping and falsifying hypotheses, a bit too little connected to their ongoing study, and perhaps welcome more general critical thinking. Thus, to paraphrase the students’ feedback (both anonymous and non-anonymous), teaching basic office software is useless because they know it well; scientific thinking is useless because they do not see a direct practical application; and presentation skills are very useful and should be allocated more hours. As a reaction, we substantially changed over half of the soft-skill seminar curriculum in the first term, decreased the timeshare allocated for general and office software packages, and allocated much more time for an introduction to specialized software tools (such as advanced graphs, basic data analysis, basic statistics, geographic information systems, infographics, biological and chemical objects, structures). However, we decided not to remove unpopular topics such as scientific thinking and instead focused on improving their attractiveness and communicating their usefulness.

Feedback as perceived by students and from a long-term perspective

Long-term outlook

The given examples of feedback and responses are illustrative of the fact that we view the SP(s) not as a permanent curriculum but rather as a process (Fig.  3 ). This is particularly important for emerging fields and fields undergoing rapid changes (such as environmental sciences and health). However, most of the students’ feedback (including the described examples) confronts, comments, or questions courses and topics with regard to short-term usefulness. The overall philosophy of the SP is considered very rarely. In addition, one of the very typical questions is “What will be my job?” The options start surfacing only during the last year of the bachelor SP or even during the following master’s program, which we found to be discouraging to students, and they often expect training for a job market from the first year.

Conceptual viewpoint on a study program as a process of improvement. Stakeholders of higher education usually have different views on the ideal SP and only one (an example) is shown for clarity. Personal interests may include anyone related to the SP, e.g., students, teachers, or administrators, and their interests do not fully overlap

This correlates well with the question of the role of education, as discussed by Liessmann (Liessmann 2008 ). Commercial and, unfortunately, often also societal demand for education is to be primarily training for a role (e.g., a job type), not for the life of alumnus and future of the society, or simply as a value in itself. Often one of the most important parameters of a program’s success is the number of alumni that managed to find a job “in their field.” In natural sciences, that usually means focusing on the quality of hard science and practical skills. Improving society by creating an intellectual elite capable of tackling future complex issues that combine scientific, social, philosophical, economic, or political aspects is not an easily quantifiable aim and thus practically not an aim at all. However, complex issues, such as those in the environment and health, need such a complex approach.

To wrap up, we see value in students’ feedback on the SP curriculum, courses, and teachers. From the point of view of SP management, it is one of the quickest types of feedback and allows testing of solutions to reported issues and creating short feedback loops. Feedback processes are often difficult to manage, and feedback literacy should be cultivated on both students’ and teachers’ sides (Carless and Winstone 2023 ). They are an important element of students’ learning (Clynes and Raftery 2008 ) and a transferrable skill by itself.

Conclusions and future remarks

Here we describe the formation and evolution of the curricula of the study programs Environment and Health. The role of feedback of various types and origins in these processes underlines the view of higher education study programs and curricula as a process. New specializations are now introduced in the discussed study programs, and they are partially based on the breadth of short- and medium-term feedback we have been obtaining. We will keep receiving and actively seeking feedback to keep the program alive and up to date.

Balaban AT, Klein DJ (2006) Is chemistry “TheCentral Science”? How are different sciences related? Co-citations, reductionism, emergence, and posets. Scientometrics 69:615–637. https://doi.org/10.1007/s11192-006-0173-2

Article   CAS   Google Scholar  

Barouki R, Audouze K, Becker C et al (2022) The exposome and toxicology: a win–win collaboration. Toxicol Sci 186:1–11. https://doi.org/10.1093/toxsci/kfab149

Benneworth P, Jongbloed BW (2010) Who matters to universities? A stakeholder perspective on humanities, arts and social sciences valorisation. High Educ 59:567–588. https://doi.org/10.1007/s10734-009-9265-2

Article   Google Scholar  

Brown TE, Bursten BE, Woodward P (2022) Chemistry: the central science, 15th edn. Pearson, London

Carless D, Winstone N (2023) Teacher feedback literacy and its interplay with student feedback literacy. Teach High Educ 28:150–163. https://doi.org/10.1080/13562517.2020.1782372

Carson R (1962) Silent spring, special edition for consumer reports subscribers only. Houghton Mifflin, Boston

Clynes MP, Raftery SEC (2008) Feedback: an essential element of student learning in clinical practice. Nurse Educ Pract 8:405–411. https://doi.org/10.1016/j.nepr.2008.02.003

Destoumieux-Garzón D, Mavingui P, Boetsch G et al (2018) The one health concept: 10 years old and a long road ahead. Front Vet Sci 5. https://doi.org/10.3389/fvets.2018.00014

European Commission (2020) European Skills Agenda - employment, social affairs & inclusion - European Commission. https://ec.europa.eu/social/main.jsp?catId=1223&langId=en . Accessed 3 Jan 2024

European Commission (2023) Horizon Europe - European Commission. https://research-and-innovation.ec.europa.eu/funding/funding-opportunities/funding-programmes-and-open-calls/horizon-europe_en . Accessed 3 Jan 2024

European Commission, Directorate-general for education Y sport and culture (2015) ECTS users’ guide 2015. Publications Office of the European Union. https://data.europa.eu/doi/10.2766/87192

Gao P (2021) The exposome in the era of one health. Environ Sci Technol 55:2790–2799. https://doi.org/10.1021/acs.est.0c07033

Jain V, Mogaji E, Sharma H, Babbili AS (2022) A multi-stakeholder perspective of relationship marketing in higher education institutions. J Mark Higher Educ 1–19. https://doi.org/10.1080/08841241.2022.2034201

Jansson S, Söderström H, Andersson PL, Nording ML (2015) Implementation of problem-based learning in environmental chemistry. J Chem Educ 92:2080–2086. https://doi.org/10.1021/ed500970y

Kuhn TS, Hacking I (2012) The structure of scientific revolutions: 50th anniversary edition, 4th edn. University of Chicago Press, Chicago

Book   Google Scholar  

Lammel G, Comas EJ, Ivancev-Tumbas I (2014) Higher education in environmental sciences with chemistry emphasis: bachelor and master programmes in Europe. Environ Sci Pollut Res 21:7211–7218. https://doi.org/10.1007/s11356-014-2737-7

Liessmann KP (2008) Theorie der Unbildung: Die Irrtümer der Wissensgesellschaft, 13th edn. Piper Taschenbuch, Munich

Malecka B, Boud D, Tai J, Ajjawi R (2022) Navigating feedback practices across learning contexts: implications for feedback literacy. Assess Eval High Educ 47:1330–1344. https://doi.org/10.1080/02602938.2022.2041544

Price EJ, Vitale CM, Miller GW et al (2022) Merging the exposome into an integrated framework for “omics” sciences. iScience 25. https://doi.org/10.1016/j.isci.2022.103976

Schwabe CW (1984) Veterinary medicine and human health, subsequent. Williams & Wilkins, Baltimore

Google Scholar  

Stone R (2002) Counting the cost of London’s killer smog. Science 298:2106–2107. https://doi.org/10.1126/science.298.5601.2106b

United Nations (2015) A/RES/70/1 Transforming our world: the 2030 agenda for sustainable development. https://www.un.org/en/development/desa/population/migration/generalassembly/docs/globalcompact/A_RES_70_1_E.pdf

Whittle DB (2016) How feedback loops can improve aid and governance. In: Thomas EA (ed) Broken pumps and promises: incentivizing impact in environmental health. Springer International Publishing, Cham, pp 31–45

Chapter   Google Scholar  

Zinsstag J, Schelling E, Waltner-Toews D, Tanner M (2011) From “one medicine” to “one health” and systemic approaches to health and well-being. Prev Vet Med 101:148–156. https://doi.org/10.1016/j.prevetmed.2010.07.003

Download references

Acknowledgements

We would like to thank all people participating in pedagogical activities at RECETOX, teams preparing the curricula, guarantors, faculties and lecturers, the study programs board, and especially all active students and alumni who are helping to shape the program. We thank Lucie Janků for critical discussions on this concept.

Open access publishing supported by the National Technical Library in Prague. This study was supported by the EU Horizon 2020 Research and Innovation Program No. 857560. The authors thank the RECETOX Research Infrastructure (No LM2023069) financed by the Ministry of Education, Youth, and Sports for providing a supportive background.

Author information

Authors and affiliations.

RECETOX, Faculty of Science, Masaryk University, Kamenice 5, 625 00, Brno, Czech Republic

Peter Šebej & Jakub Urík

You can also search for this author in PubMed   Google Scholar

Contributions

P.Š. conceptualized work with inputs from J.U. All authors participated in developing ideas, research, analysis, and writing and approved the final version of the manuscript.

Corresponding author

Correspondence to Peter Šebej .

Ethics declarations

Ethical approval.

Not applicable.

Consent to participate

Consent for publication, conflict of interest.

The authors declare no competing interests.

Additional information

Responsible Editor: Philippe Garrigues

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This publication reflects only the authors’ views, and the European Commission is not responsible for any use that may be made of the information it contains.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 85 KB)

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Šebej, P., Urík, J. Evolution of environmental chemistry study program curricula in tertiary education: a case study and general implications. Environ Sci Pollut Res (2024). https://doi.org/10.1007/s11356-024-33756-2

Download citation

Received : 08 February 2024

Accepted : 17 May 2024

Published : 01 June 2024

DOI : https://doi.org/10.1007/s11356-024-33756-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Environmental chemistry
• University education
• Environment
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 30 May 2024

Utility of a virtual small group cognitive behaviour program for autistic children during the pandemic: evidence from a community-based implementation study

• Vivian Lee 1 ,
• Nisha Vashi 2 ,
• Flora Roudbarani 2 ,
• Paula Tablon Modica 2 ,
• Ava Pouyandeh 2 ,
• Teresa Sellitto 2 ,
• Alaa Ibrahim 2 ,
• Stephanie H. Ameis 3 , 4 ,
• Alex Elkader 5 ,
• Kylie M. Gray 6 ,
• Connor M. Kerns 7 ,
• Meng-Chuan Lai 3 , 4 ,
• Johanna Lake 3 , 4 ,
• Kendra Thomson 8 , 9 &
• Jonathan A. Weiss 2  

BMC Health Services Research volume  24 , Article number:  685 ( 2024 ) Cite this article

Metrics details

Autistic children often experience socioemotional difficulties relating to emotion regulation and mental health problems. Supports for autistic children involve the use of adapted interventions that target emotion regulation and social skills, alongside mental health symptoms. The Secret Agent Society Small Group (SAS: SG), an adapted cognitive behavioural program, has demonstrated efficacy through lab-delivered randomized control trials. However, research is still needed on its effectiveness when delivered by publicly funded, community-based autism providers under real-world ecologically valid conditions, especially within the context of a pandemic. The COVID-19 pandemic has disrupted access to community-based supports and services for autistic children, and programs have adapted their services to online platforms. However, questions remain about the feasibility and clinical utility of evidence-based interventions and services delivered virtually in community-based settings.

The 9-week SAS: SG program was delivered virtually by seven community-based autism service providers during 2020–2021. The program included the use of computer-based games, role-playing tasks, and home missions. Caregivers completed surveys at three timepoints: pre-, post-intervention, and after a 3-month follow-up session. Surveys assessed caregivers’ perception of the program’s acceptability and level of satisfaction, as well as their child’s social and emotional regulation skills and related mental health challenges.

A total of 77 caregivers (94% gender identity females; Mean  = 42.1 years, SD  = 6.5 years) and their children (79% gender identity males; Mean  = 9.9 years, SD  = 1.3 years) completed the SAS: SG program. Caregivers agreed that the program was acceptable (95%) and were highly satisfied (90%). Caregivers reported significant reduction in their child’s emotion reactivity from pre- to post-intervention (-1.78 (95% CI, -3.20 to -0.29), p  = 0.01, d =  0.36), that continued to decrease after the 3-month booster session (-1.75 (95% CI, -3.34 to -0.16), p  = 0.02, d =  0.33). Similarly, improvements in anxiety symptoms were observed (3.05 (95% CI, 0.72 to 5.36), p  = 0.006, d  = 0.39).

Conclusions

As online delivery of interventions for autistic children remains popular past the pandemic, our findings shed light on future considerations for community-based services, including therapists and agency leaders, on how best to tailor and optimally deliver virtually based programming.

Trial registration

This study has been registered with ISRCTN Registry (ISRCTN98068608) on 15/09/2023. The study was retroactively registered.

Peer Review reports

Introduction

Autistic children often experience difficulties with emotion regulation and social communication skills, which can interfere with their functioning and have a negative impact on their quality of life and well-being. Difficulties in emotion regulation (i.e., challenges in monitoring, evaluating, and expressing one’s own emotions [ 16 ]) are considered transdiagnostic symptoms [ 1 , 37 ] in that they are implicated in the development of many different mental health problems, including anxiety, depression, eating disorders, and substance use [ 18 ]. Emotion regulation is also often relational in nature [ 16 ], and in autistic children, challenges with emotion regulation have been correlated with greater social communication difficulties [ 25 ]. Pandemic-related policies (e.g., closure of schools and community-based services, lockdowns, etc.) meant to limit the spread of COVID-19 likely exacerbated the emotion regulation problems, as well as social and mental health difficulties experienced by many autistic children [ 23 , 24 , 27 , 28 , 36 , 39 ].

For verbally able autistic children, variations of adapted cognitive behaviour therapy (CBT) programs have been used to improve emotion regulation skills and social skills, alongside mental health problems. For instance, work from Wood and colleagues [ 44 ] demonstrated the effectiveness of the Behavioral Interventions for Anxiety in Children with Autism (BIACA), an intervention delivered in modular format that allows social skills to be targeted alongside coping skills for anxiety. The study found that BIACA was more effective in increasing social communication skills when compared to traditional CBT programs that focused on anxiety reduction alone. Similarly, White et al. [ 40 , 42 ] demonstrated the feasibility and clinical utility of the Multimodal Anxiety and Social Skills Intervention (MASSI), an adapted CBT program that considers the interconnectedness of anxiety and social communicative challenges in autistic children. Beyond solely treating anxiety, group-based CBT programs have also been successfully adapted to target emotion regulation and social skills [ 7 , 22 , 32 , 35 ]. A randomized control trial of a one-on-one CBT program, the Secret Agent Society: Operation Regulation (SAS: OR) [ 3 ], showed improvements in emotion regulation and adaptive skills, and reductions in externalizing symptoms and overall psychiatric symptom severity [ 38 ].

Pandemic disruptions have accelerated the need for programs that leverage existing online platforms to deliver therapeutic interventions, including using synchronous (real-time) and asynchronous (recorded) sessions, homework assignments, and peer support [ 2 , 17 ]. Even before the pandemic, emerging evidence supported the effectiveness of online-based programs. For example, Beaumont and colleagues [ 6 ] conducted a pilot randomized control trial of an online version of the Secret Agent Society Small Group (SAS: SG) program [ 4 , 33 ] for autistic children within a university-setting and found improvements in parent-reported social skills and problem behaviours compared to a control group. Lee and colleagues [ 21 ] conducted a mixed-methods evaluation of an online SAS: OR program during the first wave of the pandemic and demonstrated improvements in emotion regulation, social skills, and reductions in children’s externalizing behaviours after participation in the intervention. Other programs that target social skills and anxiety were also quickly adapted for online delivery, and preliminary results demonstrated general improvements in target behaviours (PEERS - Lee et al., 2023 [ 20 ]; Facing Your Fears - McMorris et al., in prep). Although the results of these pilot programs are promising, there is still a need to explore considerations for delivering virtual programming, particularly in community-based settings where autistic children receive most of their supports.

In Canada, community-based agencies are often publicly funded and provide the bulk of services for autistic children (e.g., behavioural interventions and supports, family workshops, parent respite, core clinical services, etc.). During the pandemic, many of these agencies continued to provide adapted virtual supports (e.g., phone consultations, online programming, etc.) for families. Group programs that are delivered by community agencies have been particularly impacted by the pandemic, as lockdowns and social distancing measures limited the availability of services [ 21 , 29 ]. There is some research suggesting that in-person community agencies were among the first to close and one of the last to re-open following pandemic restrictions in Canada [ 45 ], relative to hospital or school-based programs (Data from the Canadian Institute for Health Information, see https://www.cihi.ca/en for more information).

Online delivery of programs by community services can be beneficial and help address logistical barriers that many families face [ 11 , 27 ]. Online platforms may enhance intervention adherence and accessibility, as participants can access services from their homes, reducing barriers related to transportation, resources, and time [ 6 , 11 , 23 ]. Such interventions can also be tailored to the unique needs and preferences of families, including the ability to access services outside geographical location or service boundaries (e.g., catchment area) and to participate in sessions without leaving their home [ 2 , 6 , 23 ]. Yet changes to evidence-based interventions for online delivery, especially within the context of a pandemic, require careful considerations of feasibility and intervention clinical utility.

The current study reports on the feasibility and clinical utility of an adapted virtual socioemotional intervention (SAS: SG) delivered during the pandemic by seven community agencies in Ontario, Canada. Using an effectiveness-implementation hybrid design [ 10 ], which takes a dual focus by testing the effects of a clinical intervention on relevant participant outcomes while gathering information on implementation. For this study we tested the effectiveness of participation on child socioemotional and clinical outcomes (i.e., parent-reported changes in emotion regulation and social skills, and symptoms of anxiety and depression) post-intervention and after a 3-month follow-up session. At the same time, we gathered information on the feasibility of the program’s delivery by assessing the level of intervention acceptability reported by families, session attendance, therapist fidelity, and parent ratings of intervention acceptability and satisfaction.

Participants

Families were eligible to participate in the intervention if: (a) their child was between 8 and 12 years of age; (b) the child had a confirmed autism diagnosis from a regulated healthcare professional; (c) caregivers informally reported child difficulties with emotion regulation and social functioning, and/or were waiting for supports to address emotion regulation and social skills; and (d) a caregiver was able to participate in the program. Families were excluded if the child had (a) an intellectual disability; (b) a diagnosis of acute psychosis or conduct disorder; or (c) any behaviours that made online group participation a safety concern (e.g., self-harm behaviours, etc.).

A total of 87 families, across 7 agencies, participated in the study. Ten did not complete the intervention (see results section for more information about non-completing families). Of the 77 primary caregivers (94% mothers; Mean age = 42.5 years, SD age = 5.7 years) who completed the program, 67 completed the optional 3-month follow-up booster session. Caregivers identified as primarily White (72%), South/West/East Asian (12%), multiethnic (5%), Latin American/Hispanic (5%), and Black (2%). Children (79% identified their gender as males; Mean age = 9.9 years, SD age = 1.3 years) identified as White (66%), multiethnic (17%), South/West/East Asian (9%), Black (2%) and Latin American/Hispanic (2%). Additional participant characteristics are presented in Table  1 .

The study was approved by the research ethics board at the researchers’ institution, an academic hospital, and by the research review committee at two community-based agencies. The project was supported by a community-partner participatory framework [ 19 ], and was co-designed by a team of researchers, as well as with community agency leadership and frontline staff (e.g., therapists, child, and youth workers, etc.). Prior to implementation, researchers met with agencies to discuss agency-specific recruitment strategies and protocols, and how best to incorporate the intervention into existing programming without interfering with overall service deliverables.

Seven community-based autism service providers across Southern Ontario participated in the implementation of the SAS: SG project between October 2020 and December 2021. Prior to the delivery of each group, 21 therapists participated in a standardized four-day online training in August 2020 facilitated by the SAS: SG development team. Please see the Appendix to review therapist demographics including their level of education and discipline of practice.

Families were screened and recruited by each agency. Agencies followed their usual screening and enrollment protocol, as outlined by their own agency guidelines and policies, for offering services to children and their families on their client list. In publicly funded service providing agencies, children only require an autism diagnosis to get access to services and supports, and do not have to meet clinical cut-offs to enroll in interventions targeting emotion regulation and social skills. Therapists will use clinical judgement to determine which programs would best match the child’s and/or family needs. In some agencies, caregivers can self-refer their child if they feel that the focus of a program might be a good fit for their child. Some participants were recruited internally from agency waitlists, and some agencies recruited participants using social media posts or emailing past clients. All families were screened according to the inclusion and exclusion criteria. Once a family was deemed a good fit for the program by the SAS: SG therapist team (e.g., ready to receive intervention, available for group sessions, family goals align with program targets, etc.), researchers contacted the participants to review the research consent and provide details for participation in the study. Caregivers were then sent the pre-intervention child and family measures to be completed online. Caregivers completed post-intervention child and family measures (see below for program description and delivery schedule), and again after the 3-month follow-up booster session.

Intervention

The Secret Agent Society: Small Group Program (SAS: SG; Social Science Translated) [ 3 – 5 ] is a spy-themed manualized cognitive behavioural program focused on helping school-age children with identified emotion regulation and social skill difficulties. All caregiver and child sessions were delivered virtually through Zoom or Microsoft Teams. The program included separate caregiver and child sessions facilitated by therapists from each agency, as well as between-session practice activities and inclusive classroom tip sheets for each child’s schoolteachers. Agencies had the option to deliver the parent and child group sessions at the same time, or on different days, but the modules were synchronized to ensure that the parent session reviewed concepts covered in the child group sessions. Child sessions targeted social communication skills, working on teams, problem solving, developing and maintaining friendships, recognizing emotions in oneself and others, coping with feelings of anger and anxiety, and expressing emotions in helpful ways (for more specific information about the intervention, see https://www.secretagentsociety.com/ ). The child sessions were either provided as a weekly 9-session (90 min per session) or an 18-session (45 min per session) format. In the current study, out of 77 children, 65 (84.4%) received the 9-session format and 12 (15.6%) received the 18-session format. In the 9-session format, 92.3% attended at least 8 sessions or more, and in the 18-session format, 83% attended at least 16 sessions or more. The sessions were facilitated by either one or two trained facilitators, with a group of 3–4 or 4–6 children.

Caregiver sessions reviewed key components from the child sessions and teach caregivers how to support generalization of skills at home and beyond. Caregiver sessions were delivered in three different formats, and agencies could choose the schedule that worked best for them. The formats included (1) 9 weekly sessions of 45 min per week; (2) 18 weekly sessions of 30 min per week; or (3) three 2-hour sessions every 3 weeks. All agencies offered a 2-hour parent information session prior to beginning the program. In the current study, 56 (72.7%) caregivers received the 9 sessions module, 12 (15.6%) caregivers received the 18 sessions module, and 9 (11.7%) received the three 2-hour sessions. In the 9-session format, 96.4% attended at least 8 sessions or more; in the 18-session format, 83% attended at least 16 sessions or more; and in the 3-session format, 55.5% attended at least 2 sessions or more.

Implementation measures

Attendance. Therapists tracked attendance for the weekly sessions and the 3-month booster session.

Fidelity . Therapists tracked their adherence to the SAS: SG protocol using a weekly session checklist. The checklists were collected after completion of the program, and fidelity was calculated as the percentage of completed tasks across all sessions.

Implementation Acceptability Scale (IAS) [ 23 ]. The IAS is a 7-item lab-developed measure to assess intervention acceptability at the end of the 9-week sessions, based on Sekhon and colleagues’ theoretical framework of acceptability [ 31 ]. Caregivers were asked to describe their experience receiving the intervention using a five-point Likert scale (1 = “ strongly disagree ” to 5 = “ strongly agree ”), with higher scores reflecting greater treatment acceptability. Caregivers rated various dimensions, including affective attitude, burden, ethicality, intervention coherence, opportunity costs, perceived effectiveness, and self-efficacy. We evaluated acceptability based on the percentage of respondents that at least indicated “ agreed ” or higher (e.g., 3 or higher on the scale) for each question.

Program Satisfaction Questionnaire (PSQ) [ 3 ]. Caregivers completed the PSQ, which assessed their views on the appropriateness and effectiveness of the program. Open-ended questions asked caregivers to comment on changes in their child’s skills or behaviour, confidence in supporting their child, enjoyment of the program, and satisfaction with the therapists. Caregivers were also asked to describe their satisfaction with different components of the program on a five-point Likert scale (0 = “ not at all satisfied ” to 5 = “ very satisfied ”). The program components included: format, session dates and times, number of sessions and session length, and overall program satisfaction.

Child outcome measures

The Social Responsiveness Scale 2nd Edition (SRS-2) [ 9 ]. The SRS-2 is a 65-item caregiver-report measure used to capture school-aged (4–18-year-olds) children’s social functioning and autism-related characteristics. Caregivers are asked to respond on a 4-point Likert Scale (0 = “ Not True ” to 3 = “ Almost Always True ”) to statements related to their child’s social functioning including in areas of social awareness, social cognition, social communication, social motivation, and the presence of restricted interests and repetitive behaviours. The SRS-2 has good external reliability ( 0.90), with strong internal consistency [ 9 ]. Additionally, this measure has high predictive validity (0.92) and construct validity [ 9 ]. It is one of the most widely used measures of children’s actual social performance and it can be expected to show moderate to large changes in the context of a successful clinical intervention [ 43 ]. In the current study, the Total T-score, Social Communication and Interaction (SCI) scale T-score and the Restricted Interests and Repetitive Behavior (RRB) scale T-score were used.

The Emotion Dysregulation Inventory (EDI) [ 26 ]. The EDI is a caregiver-report measure developed to assess the severity of autistic children’s struggles with negative mood and reactivity via two subscales, Dysphoria (6 items; anhedonia, sadness, nervousness) and Reactivity (7 items; explosive outbursts, difficulty calming, rapid escalation, intense/extreme/inappropriate emotionality). Items are rated on a 5-point Likert scale (0 = “not at all” to 4 = “very severe” ). The items were summed for each subscale and converted to T - Scores. For both subscales, higher scores indicated greater dysregulation. The EDI shows strong validity and reliability for assessing Reactivity and Dysphoria in autistic children [ 26 ]. Internal consistency for Dysphoria and Reactivity within the current sample pre-intervention were very good: α = 0.88 and α = 0.89, respectively.

Child and Adolescent Symptoms Inventory-5 ( CASI-5) [ 13 ]. The CASI-5 is a caregiver-report measure that gathers information about the symptoms of Diagnostic and Statistical Manual and Mental Disorders- 5th Edition (DSM-5) defined disorders in children and adolescents between the ages of 5 to 18 years. The 173-item inventory is organized into modules where each consists of a list of symptom statements for 14 of the most commonly defined DSM-5 disorders. Caregivers are asked to rate whether their child displays any of the symptoms on a 4-point scale ranging from “ never ” to “ very often ”. The tool shows strong validity and reliability in caregivers of autistic children, including overlap with interview measures of mental health disorders [ 12 ]. This measure has been found to have very good internal consistency for assessments of anxiety (α = 0.85-0.88) [ 14 ] and depression (α = 0.83) [ 15 ] in parent-reports. In the current study, we used the total symptom severity T-Score for separation anxiety, social anxiety, generalized anxiety disorder, and major depression disorder, with higher scores indicating greater level of presenting symptoms.

Data analyses

All analyses were conducted using SPSS version 28. Implementation acceptability and feasibility were explored using descriptive statistics, while changes in child outcome measures were analyzed using paired t-tests and repeated-measures ANOVAs, with a Greenhouse-Geisser correction to account for violations of sphericity. Post hoc analyses of ANOVA outcomes used Bonferroni corrections.

Non-completer participant profiles

A preliminary evaluation of the demographic profiles and key baseline characteristics of families included those who did not complete the program are outlined in Table  2 ( n  = 10). Families listed various reasons for being unable to continue with the program including scheduling issues with the group sessions ( n  = 2), the program required too much time commitment ( n  = 2), lack of interest in the theme ( n  = 1), virtual format not a good fit for their child ( n  = 3), and urgent family obligations ( n  = 2). These caregivers attended on average 2.43 sessions ( Range  = 1–3) and children attended 1.67 sessions ( Range  = 0–4). There were no significant differences in age or gender distributions between non-completer and completer caregivers. Independent sample t-tests showed a significantly higher SRS-2 RRB T-score for non-completer children ( M  = 78.3, SD  = 9.60) compared to completer children ( M  = 78.30, SD  = 9.03, F (1,86) = 4.14, p  = 0.04). There were no other significant differences in pre-intervention child outcome measures or child demographics.

Implementation results

As shown in Table  3 , almost all treatment completers attended their weekly parent and child group sessions (attended 93.8% of sessions).

On the self-reported checklists, therapists indicated above 80% fidelity for both caregiver ( M  = 93.5%, Range  = 88–99%) and child ( M  = 86.6%, Range  = 78.9–6.9%) weekly sessions. A review of the fidelity checklists suggested that therapists were unable to complete some parts because of technology issues that prevented the completion of certain activities (e.g., online board game, virtual missions with the group, poor internet connections preventing participation, etc.), ran out of time to do an activity during the session (which resulted in assigning the task as homework), and/or unexpected disruptions (e.g., child abruptly disengages from the group, home-based interference, etc.). In terms of post-program acceptability, 75% of caregivers agreed or strongly agreed that they felt positively about the program, 95% agreed or strongly agreed that it aligned with their values, 87% agreed that they understood how it worked, 77% agreed that they did not have to give up resources or opportunities to participate in the program, and 77% agreed that they felt confident in the skills they had learned. Lower ratings of acceptability related to acceptable amount of effort to participate (only 61% agreed) and feeling that it was effective in achieving its goals (only 62% agreed). A qualitative analysis of caregiver feedback ( n  = 20) indicated that the virtual format required parents to spend more time monitoring their child’s group sessions in order to manage their behaviours, and to help them stay engaged. Some caregivers ( n  = 10) hoped that participation would lead to new emotion regulation or social skills but instead were somewhat disappointed when the program only reinforced their child’s current skill set. On the post-intervention PSQ, 70% of the caregivers reported feeling “ moderately ” to “ very ” confident in their ability to support their child’s future social and emotional development following completion of the program, and 83% reported that the program was “ moderately ” to “ very ” enjoyable for their child. Caregivers reported being “ moderately ” to “ very ” satisfied (90%) with their SAS: SG group facilitator, and overall, 71% of caregivers reported being “ moderately ” to “ very ” satisfied with the entire program.

Child outcomes

The pre-intervention SRS-2 Total T-score ranged from 54 to 90 ( M  = 71.33, SD  = 8.91). For the SCI scale, the pre-intervention t-score ranged from 54 to 90 ( M  = 70.52, SD  = 9.07) and the RRB scale T-score ranged from 52 to 90 ( M  = 72.03, SD  = 9.28). In our sample, 91% of children met clinical level of concern on the SRS-2 Total T-Score. As shown in Table  4 , there was a significant difference between SRS-2 Total T-Scores, SCI, and RRB T-scores across pre-, post-, and the 3-month time points. Scores consistently decreased over time, which demonstrated improvements from pre- to post-intervention (Mean Difference = -3.14 (95% CI, -7.86 to -4.60, p  = 0.001, d  = 0.55)), and from post-intervention to the 3-month booster session (Mean Difference = -2.76 (95% CI, -5.03 to -0.49, p  = 0.001, d  = 0.52)) on the SRS-2 Total T-score.

On the emotion dysregulation measure (EDI), there were significant differences between time points on the EDI Reactivity and Dysphoria T-scores. The pre-intervention EDI Reactivity T-scores ranged from 30.1 to 66.7 ( M  = 50.13, SD  = 7.07) and the EDI Dysphoria T-score ranged from 36.4 to 70.3 ( M  = 48.01, SD  = 8.84), with 52% and 25% of the children in our sample meeting clinical cut-offs for emotion regulation difficulties across the two scales, respectively [ 8 ]. EDI Reactivity scores decreased from pre-intervention to post-intervention (-1.78 (95% CI, -3.2 to -0.29), p  = 0.01, d =  0.36), and continued to decrease from post-intervention to after the 3-month booster session (-1.75 (95% CI, -3.34 to -0.16), p  = 0.02, d =  0.33). While EDI Dysphoria scores decreased from pre-intervention to post-intervention, this change was not significant. EDI Dysphoria scores continued to decrease after the 3-month booster, with scores being significantly lower than the pre-intervention scores (-3.13 (95% CI, -5.02 to -1.23), p  = 0.001, d =  0.51) but not the post-intervention scores (-1.67 (95% CI, -3.40 to 0.56, p  = 0.061).

Scores on the CASI-5 indicated that pre-intervention, 38% of the children met clinical range of concerns for separation anxiety ( T -score range = 50.0–78.0), 49% for social anxiety ( T -score range = 50.0–76.0), 83% for general anxiety disorder ( T -score range = 50.0–78.0), and 43% for depression ( T -score range = 50.0–78.0). There were significant changes across time points with respect to symptoms related to both generalized anxiety disorder (GAD) and major depression, but not for separation anxiety and social anxiety. GAD symptom severity scores improved from pre-intervention to post-intervention (3.05 (95% CI, 0.72 to 5.36), p  = 0.006, d  = 0.39), and then remained stable from post-intervention to the 3-month booster session (0.80 (95% CI (-3.38 to 1.77), p  = 1.00). For depression, there appeared to be no statistically significant change from pre- to post-intervention (2.29 (95% CI, -0.56 to 5.16), p  = 0.16), but there was a significant improvement from pre-intervention to the 3-month booster session (3.61 (95% CI (0.84 to 6.38), p  = 0.006, d =  0.39) (See Table  4 ).

The present study used an effectiveness-implementation hybrid design to evaluate the effects of an adapted virtual cognitive behaviour program, SAS: SG, on autistic children’s socioemotional outcomes while collecting information on community-based implementation. The SAS: SG program is publicly available, but existing research has largely focused on outcomes from lab- or university-based evaluations, and prior to the COVID-19 pandemic, it was delivered primarily in an in-person format. Findings from the current study suggest that the program was implemented successfully with high therapist-reported fidelity across seven community autism-focused service agencies and provides support for more rigorous research into the efficacy of group-based online programs for autistic children in the community. Results from our study suggest that families completed most of their weekly parent and child group sessions, with similar attendance rates compared to in-person adapted group programs [ 44 ] and other online versions of the program [ 23 ]. The program also saw a rather low attrition rate ( ∼  10%) which may have reflected the strengths of an online formatting that decreased the usual barriers to participation including the cost of travel (e.g., time and financial costs).

In terms of feasibility, caregivers rated most aspects of intervention acceptability as high and described feeling positively about the program, that it aligned with their values, and that they understood how the program worked. However, it should be noted that a substantial group of caregivers did not agree that the program demanded a reasonable amount of effort from them (39%) or that it was effective in achieving its goals (38%). A review of text-based comments from caregivers indicated that the amount of time that was asked of them was sometimes overwhelming, including having to monitor their child’s participation, learn new concepts, support their child’s learning of skills, and facilitate assigned home activities on a weekly basis. For some caregivers, this led to hours of work above and beyond their own participation in the parent groups. These themes are consistent with previous findings related to delivering caregiver-involved online programs during the pandemic [ 17 , 23 , 41 ]. In terms of goal achievement, some caregivers were underwhelmed by the usefulness of the skills taught in the program. A qualitative review of text responses from caregivers indicated that that some hoped that their child could have learned new emotion regulation and social skills rather than practice skills they had already mastered. Some caregivers felt that the online format did not provide enough opportunities for their children to practice and apply the social skills being taught in the program, thus not achieving their original goals.

Considering caregiver feedback about the program, it is important for virtually delivered programs to consider the demands placed on caregivers that build upon existing stressors in their life [ 21 ]. During the pandemic, this reflected the additional demands of managing online support for their children, the evolving virtual school requirements, on top of their own work and household responsibilities, COVID-related illness, or other stressors. There is literature highlighting how this added burden is often placed upon primary caregivers, usually mothers. There is an urgent need to acknowledge these considerations around equity of supports for caregivers during the pandemic and beyond [ 30 ]. A review of non-completing families suggests that children who had higher levels of RRBs had a more difficult time engaging in online sessions. This is consistent with previous work [ 23 ] suggesting that program delivery with an online format may not be well suited to all caregivers and autistic children, especially those with behaviours that interfere with sitting and attending (e.g., compulsive behaviours, self-injurious behaviours, etc.).

Although there are benefits with delivering a program online (e.g., limiting the cost of travel, enabling further research for community service providers, etc.), some families may require different supports to make participating online more accessible to them. This might include adaptations like shorter sessions, more frequent assessment of motivation and engagement, greater use of specialized interests, and adaptations that focus on individualized care. For example, Mootz et al. [ 27 ] described modifications made to optimize participation for a single group pilot SAS: SG program delivered via telehealth in Australia during the pandemic. They described similar needs to develop procedures to support families including troubleshooting technology throughout delivery, shortening sessions, and tasking caregivers with supervision of child sessions (e.g., giving out end-of-session rewards, specifying consequences for non-engagement, etc.). Yet, despite online adaptations some children and their families may still find in person programming more beneficial and better suited to their needs. Future research by the current team includes a direct comparison of in person versus online version of the SAS: SG program in community-based services.

It should be noted that the current study took place at the beginning of the pandemic when rolling lockdowns were prevalent and there were few competing activities for families that required travel (e.g., other appointments, recreational activities, etc.). During this time, some caregivers were actively seeking access to any programs for their children, which may have contributed to the rather high engagement with the virtual program (e.g., lower than usual attrition rate). In addition, reduced demands associated with online delivery of the program (i.e., less travel time) may have contributed to increased feasibility and satisfaction with the program.

Successful implementation of this program may be the result of a community-partnered participatory framework that allowed each agency flexibility in recruitment and delivery [ 19 ]. Agencies managed their own scheduling (e.g., number of weeks, days, and times, etc.), and were supported by the research team throughout the project (e.g., troubleshooting technology issues, etc.). Therapist-reported fidelity suggested that adherence was high (87% or higher), although we could not independently verify their session fidelity as sessions were not recorded. Therapists did face some challenges completing parts of the modules, especially during child sessions, due to technology issues, running out of time, and unexpected disruptions. These are important factors to consider from an implementation perspective for future hybrid service delivery. Anecdotally, feedback from therapist teams suggests that those who spent more time troubleshooting and preparing for technological issues were able to react and respond better when issues arose during program delivery. Some therapists noted that caregiver involvement was necessary to deescalate emotionally tense situations with their children, which, as noted, often increased the demands placed upon caregivers. Therapists mentioned that specialized interests were incorporated into group sessions as necessary, and overall, most worked hard (e.g., provided visual aids, used animations, and used an abundance of reinforcers and/or tokens, etc.) to engage the children in the group sessions. Therapists should consider individual child and family needs, and screening participants for suitability for online-based group programs should consider access to technology and a family’s ability to support their child’s participation in the program [ 13 , 23 ].

In terms of clinical utility of the program, caregivers reported improvements in child emotion regulation and social communication skills from pre- to post-intervention, and these gains were sustained after the 3-month booster sessions. Caregivers also reported that their children showed statistically significant improvements in social interactions and communication behaviours, and emotion reactivity at each time point. Emotion dysphoria, however, revealed a different pattern, as scores did not show a statistically significant improvement until after the 3-month booster. These findings are consistent with the main tenets of the SAS: SG program which aims to teach children deescalating techniques to prevent emotionally reactive behaviours in the face of intense or socially frustrating situations, and these skills may take additional time to solidify.

Consistent with previous work [ 17 ], no changes were found in separation and social anxiety at post-intervention and following the 3-month booster session. Given that many children were at home during the early waves of the pandemic, they may have had fewer opportunities to socialize, or be separated from caregivers. Caregivers did report improvements in children’s generalized anxiety symptoms, even after the 3-month booster session. Similar to the emotional dysphoria findings, caregivers reported steady improvements in symptoms of depression over time, but the scores reached statistically significant levels of improvement only after the 3-month booster session. These findings may suggest that the program may indirectly benefit dysphoric or behaviours resulting in negative moods and, with practice, symptoms improve over time, even though it does not specifically target them.

Interpretation of the results should be mindful of a few study limitations. The study was a single-arm implementation trial which makes our results particularly suspectable to placebo effects, and results were interpreted without a control group or blinded independent clinical assessments. Data from the study were based mainly on caregiver reports which may differ from therapist and child perspectives. Fidelity ratings were self-reported by therapists and could not be independently verified by recordings, and future implementation trials would benefit from independent coding of recorded sessions for reliability. Interpretation of the findings should consider the variability in symptom severity of our sample, especially since not all children met clinical levels of concern pre-intervention, on emotion dysregulation (52% for reactivity and 25% for dysphoria), and mental health symptoms (e.g., only symptoms of generalized anxiety were above clinical threshold pre-intervention, with social anxiety and depression symptoms being moderately elevated). Although our results showed post-intervention improvements across these domains with small to moderate effect sizes, they may not reflect a clinically meaningful change as expected for most interventions. Clinically meaningful improvements in symptoms related to generalized anxiety, however, were observed, suggesting that the socioemotional support program may have some indirect impact on improving some aspects of mental health. Finally, there seems to be particular risks in overinterpreting improvements post-intervention that may simply be due to family acclimations to pandemics stressors. Although, the SAS: SG program was delivered between September 2020 to September 2021, and started at least 4 months after the initial shut-downs due to the pandemic, which suggests that families had some time to adjust.

Despite these limitations, our findings suggest that an evidence-based intervention targeting emotion regulation and social skills in autistic children is feasible and can be delivered by community-based service providers with success. To our knowledge, this is the first study to demonstrate the effectiveness of an adapted, virtual group-based program focused on socioemotional skills, delivered in the community for autistic children during the early waves of the pandemic. Results highlight the need for ongoing support for autistic children, especially given the unpredictable circumstances imposed by the past and future pandemic resulting in a global loss of supports (e.g., therapy, social skills groups, academic and recreational programming). Our findings encourage community-partnerships with publicly funded agencies and contribute to the emerging efforts to narrow the gap from research to practice in implementing evidence-based programs in community settings. Ultimately, training in evidence-based programs by community providers can increase access to helpful ways of supporting emotion regulation and social challenges for autistic children.

Data availability

The datasets generated and/or analysed during the current study are not publicly available due university-imposed restrictions around data sharing agreements but are available from the corresponding author on reasonable request.

Aldao A, Nolen-Hoeksema S, Schweizer S. Emotion-regulation strategies across psychopathology: a meta-analytic review. Clin Psychol Rev. 2010;30(2):217–37. https://doi.org/10.1016/j.cpr.2009.11.004 .

Article   PubMed   Google Scholar  

Ameis SH, Lai MC, Mulsant BH, Szatmari P. Coping, fostering resilience, and driving care innovation for autistic people and their families during the COVID-19 pandemic and beyond. Mol Autism. 2020;11(1):61. https://doi.org/10.1186/s13229-020-00365-y .

Article   PubMed   PubMed Central   Google Scholar  

Beaumont R. Secret agent society: solving the mystery of social encounters—facilitator kit. Brisbane, Australia: The Social Skills Training Institute, A Subdivision of Triple P International; 2010.

Google Scholar  

Beaumont R, Hinton S, Sofronoff K. (2018). The Secret Agent Society: Upskilling teachers in the delivery of a game-based social skills children program. In D. Mentor, editor, Computer-mediated learning for workforce development. IGI Global. https://doi.org/10.4018/978-1-5225-4111-0.ch002 .

Beaumont R, Sofronoff K. A multi-component social skills intervention for children with Asperger syndrome: the Junior Detective Training Program. J Child Psychol Psychiatry. 2008;49(7):743–53. https://doi.org/10.1111/j.1469-7610.2008.01920.x .

Beaumont R, Walker H, Weiss J, Sofronoff K. Randomized controlled trial of a video gaming-based social skills program for children on the autism spectrum. J Autism Dev Disord. 2021;51(10):3637–50. https://doi.org/10.1007/s10803-020-04801-z .

Conner CM, White SW, Beck KB, Golt J, Smith IC, Mazefsky CA. Improving emotion regulation ability in autism: the emotional awareness and skills enhancement (EASE) program. Autism: Int J Res Pract. 2019;23(5):1273–87. https://doi.org/10.1177/1362361318810709 .

Article   Google Scholar  

Connor CM, Golt J, Schaffer R, Righi G, Siegel M, Mazefsky CA. Emotion dysregulation is substantially elevated in Autism compared to the General Population: impact on Psychiatric services. Autism Res. 2021;14(1):169–81.

Constantino JN, Gruber CP. Social Responsiveness Scale- Second Edition (SRS-2). Torrance, CA. Western Psychological Services; 2012.

Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. https://doi.org/10.1097/MLR.0b013e3182408812 .

Ellison KS, Guidry J, Picou P, Adenuga P, Davis TE 3rd. Telehealth and autism prior to and in the age of COVID-19: a systematic and critical review of the last decade. Clin Child Fam Psychol Rev. 2021;24(3):599–630. https://doi.org/10.1007/s10567-021-00358-0 .

Gadow KD, Sprafkin J. Child symptom inventory 4: CSI. Stony Brook, NY: Checkmate Plus; 1997.

Gadow KD, Sprafkin J. Fact sheet for the child and adolescent Symptom Inventory-5. Stony Brook, NY: © Checkmate Plus; 2013.

White SW, Schry AR, Maddox BB. Brief report: the assessment of anxiety in high-functioning adolescents with autism spectrum disorder. J Autism Dev Disord. 2012;42:1138–45.

Witwer AN, Lecavalier L, Norris M. Reliability and validity of the children’s interview for psychiatric syndromes-parent version in autism spectrum disorders. J Autism Dev Disord. 2012;42:1949–58.

Gross JJ, Thompson RA. Emotion regulation: conceptual foundations. In: Gross JJ, editor. Handbook of emotion regulation. New York, NY: Guilford Press; 2007. pp. 3–24.

Kalvin CB, Jordan RP, Rowley SN, Weis A, Wood KS, Wood JJ, Ibrahim K, Sukhodolsky DG. Conducting CBT for anxiety in children with Autism Spectrum Disorder during COVID-19 pandemic. J Autism Dev Disord. 2021;51(11):4239–47. https://doi.org/10.1007/s10803-020-04845-1 .

Kranzler A, Young JF, Hankin BL, Abela JRZ, Elias MJ, Selby EA. Emotional awareness: a transdiagnostic risk factor for internalizing symptoms in children and adolescents? J Clin Child Adolesc Psychology=. 2016;45(3):262–9. https://doi.org/10.1080/15374416.2014.987379 .

Kwon SC, Tandon SD, Islam N, Riley L, Trinh-Shevrin C. Applying a community-based participatory research framework to patient and family engagement in the development of patient-centred outcomes research and practice. Translational Behav Med. 2018;8(5):683–91. https://doi.org/10.1093/tbm/ibx026 .

Lee CM, Petricek C, Haga M, Smalley S, Pelletier K, Essa N, Hudock RL. Examining the feasibility and effectiveness of PEERS for adolescents via telehealth during the COVID-19 pandemic. Res Autism Spectr Disorders. 2023;109:102279.

Lee V, Albaum C, Tablon Modica P, Ahmad F, Gorter JW, Khanlou N, McMorris C, Lai J, Harrison C, Hedley T, Johnston P, Putterman C, Spoelstra M, Weiss JA. The impact of COVID-19 on the mental health and wellbeing of caregivers of autistic children and children: a scoping review. Autism Res. 2021;14(12):2477–94. https://doi.org/10.1002/aur.2616 .

Lee J, Lee TS, Lee S, Jang J, Yoo S, Choi Y, Park YR. Development and application of a metaverse-based social skills training program for children with autism spectrum disorder to improve social interaction: protocol for a randomized controlled trial. JMIR Res Protocols. 2022;11(6):e35960.

Lee V, Roudbarani F, Modica T, Pouyandeh P, A., Weiss JA. Adaptation of cognitive behavior therapy for autistic children during the pandemic: a mixed-methods program evaluation. Evidence-Based Pract Child Adolesc Mental Health. 2021;1–18. https://doi.org/10.1080/23794925.2021.1941432 .

Liu JJ, Bao Y, Huang X, Shi J, Lu L. Mental health considerations for children quarantined because of COVID-19. Lancet Child Adolesc Health. 2020;4(5):347–9. https://doi.org/10.1016/S2352-4642(20)30096-1 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mazefsky CA, Herrington J, Siegel M, Scarpa A, Maddox BB, Scahill L, White SW. The role of emotion regulation in autism spectrum disorder: emotion regulation in ASD. J Am Acad Child Adolesc Psychiatry. 2013;52(7):679–88. https://doi.org/10.1016/j.jaac.2013.05.006 .

Mazefsky CA, Yu L, White SW, Siegel M, Pilkonis PA. The emotion dysregulation inventory: psychometric properties and item response theory calibration in an autism spectrum disorder sample. Autism Res. 2018;11(6):928–41.

Mootz CA, Lemelman A, Giordano J, Winter J, Beaumont R. Brief report: feasibility of delivering the Secret Agent Society group social skills program via telehealth during COVID-19: a pilot exploration. J Autism Dev Disord. 2022;52(12):5274–9. https://doi.org/10.1007/s10803-022-05591-2 .

Mutluer T, Doenyas C, Aslan Genc H. Behavioral implications of the Covid-19 process for autism spectrum disorder, and individuals’ comprehension of and reactions to the pandemic conditions. Front Psychiatry. 2020;11:561882. https://doi.org/10.3389/fpsyt.2020.561882 .

Pavlopoulou G, Wood R, Papadopoulos C. Impact of Covid-19 on the experiences of caregivers and family carers of autistic children and young people in the UK. London, UK: UCL Institute of Education; 2020.

Pellicano E, Stears M. The hidden inequities of COVID-19. Autism. 2020;24(6):1309–10.

Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res. 2017;17:88. https://doi.org/10.1186/s12913-017-2031-8 .

Sofronoff K, Beaumont R, Weiss JA. (2014). Treating transdiagnostic processes in ASD: Going beyond anxiety. In T. E. Davis III, S. W. White, & T. H. Ollendick, editors, Handbook of Autism and Anxiety (pp. 171–183). Springer International Publishing. https://doi.org/10.1007/978-3-319-06796-4_12 .

Sofronoff K, Silva J, Beaumont R. The Secret Agent Society social-emotional skills program for children with a high-functioning autism spectrum disorder: a parent-directed trial. Focus Autism Other Dev Disabil. 2017;32(1):55–70. https://doi.org/10.1177/1088357615583467 .

Stahmer AC, Brookman-Frazee L, Rieth SR, Stoner JT, Feder JD, Searcy K, Wang T. Parent perceptions of an adapted evidence-based practice for toddlers with autism in a community setting. Autism. 2017;21(2):217–30. https://doi.org/10.1177/1362361316637580 .

Temkin AB, Beaumont R, Wkya K, Hariton JR, Flye BL, Sheridan E, Miranda A, Vela J, Zendegui E, Schild J, Gasparro S, Loubriel D, Damianides A, Weisman J, Silvestre A, Yadegar M, Catarozoli C, Bennett SM. (2022). Secret Agent Society: A randomized controlled trial of a transdiagnostic children social skills group treatment. Research on Child and Adolescent Psychopathology , 50 (9), 1107–1119. https://doi.org/10.1007/s10802-022-00919-z Thompson, R. A. (1994). Emotion regulation: A theme in search of definition. Monographs of the Society for Research in Child Development , 59 (2–3), 25–52.

Vasa RA, Singh V, Holingue C, Kalb LG, Jang Y, Keefer A. Psychiatric problems during the COVID-19 pandemic in children with autism spectrum disorder. Autism Res. 2021;14(10):2113–9.

Weiss JA. Transdiagnostic case conceptualization of emotional problems in children with ASD: an emotion regulation approach. Clin Psychol. 2014;21(4):331–50. https://doi.org/10.1111/cpsp.12084 .

Weiss JA, Thomson K, Burnham Riosa P, Albaum C, Chan V, Maughan A, Tablon P, Black K. A randomized waitlist-controlled trial of cognitive behavior therapy to improve emotion regulation in children with autism. J Child Psychol Psychiatry. 2018;59(11):1180–91. https://doi.org/10.1111/jcpp.12915 .

Weyland M, Maes P, Kissine M, Defresne P. (2022). Impact of Belgian COVID-19 lockdown restrictions on autistic individuals’ socio-communicative behaviors and their caregivers’ quality of life. PLoS ONE, 17(8), e0273932.

White SW, Albano AM, Johnson CR, Kasari C, Ollendick T, Klin A, Oswald D, Scahill L. Development of a cognitive-behavioral intervention program to treat anxiety and social deficits in teens with high-functioning autism. Clin Child Fam Psychol Rev. 2010;13(1):77–90. https://doi.org/10.1007/s10567-009-0062-3 .

White LC, Law JK, Daniels AM, Toroney J, Vernoia B, Xiao S, Consortium SPARK, Feliciano P, Chung WK. Brief report: impact of COVID-19 on individuals with ASD and their caregivers: a perspective from the SPARK cohort. J Autism Dev Disord. 2021;51(10):3766–73. https://doi.org/10.1007/s10803-02004816-6 .

White SW, Ollendick T, Albano AM, Oswald D, Johnson C, Southam-Gerow MA, Kim I, Scahill L. Randomized controlled trial: multimodal anxiety and social skill intervention for adolescents with autism spectrum disorder. J Autism Dev Disord. 2013;43(2):382–94. https://doi.org/10.1007/s10803012-1577-x .

Wolstencroft J, Robinson L, Srinivasan R, Kerry E, Mandy W, Skuse D. A systematic review of group social skills interventions, and meta-analysis of outcomes, for children with high functioning ASD. J Autism Dev Disord. 2018;48:2293–307.

Wood JJ, Kendall PC, Wood KS, Kerns CM, Seltzer M, Small BJ, Lewin AB, Storch EA. Cognitive behavioral treatments for anxiety in children with autism spectrum disorder: a randomized clinical trial. JAMA Psychiatry. 2020;77(5):474–83. https://doi.org/10.1001/jamapsychiatry.2019.4160 .

Yuan P, Aruffo E, Gatov E, Tan Y, Li Q, Ogden N, Collier S, Nasri BR, Moyles I, Zhu H. School and community reopening during the COVID-19 pandemic: a mathematical modelling study. Royal Soc Open Sci. 2022;9:211883. https://doi.org/10.1098/rsos.211883 .

Article   CAS   Google Scholar  

Download references

Acknowledgements

The authors would like to thank the families who participated in the groups and our partnered community agencies who co-designed and implemented this research study.

The authors disclose receipt of the following support for the research, authorship, and/or publications of this article. This study was supported by the Kids Brain Health Network, and by the York Research Chair in Autism and Neurodevelopmental Disability Mental Health.

Author information

Authors and affiliations.

Department of Psychology, Carleton University, 214E Social Science Research Building, Ottawa, ON, K1S 5B6, Canada

Department of Psychology, York University, 230 BSB, 4700 Keele St, Toronto, ON, M3J 1P3, Canada

Nisha Vashi, Flora Roudbarani, Paula Tablon Modica, Ava Pouyandeh, Teresa Sellitto, Alaa Ibrahim & Jonathan A. Weiss

Centre for Addiction and Mental Health, 1001 Queen Street West, Toronto, ON, M6J 1H4, Canada

Stephanie H. Ameis, Meng-Chuan Lai & Johanna Lake

Department of Psychiatry, Temerty Faculty of Medicine, University of Toronto, 250 College Street, 8th Floor, Toronto, ON, M5T 1R8, Canada

Kinark Child and Family Services, 7271 Warden Ave, Markham, ON, L3R 5X5, Canada

Alex Elkader

Centre for Education Development, Appraisal, and Research, University of Warwick, Coventry, CV4 7AL, UK

Kylie M. Gray

Department of Psychology, University of British Columbia, 2136 West Mall, Vancouver, BC, V6T 1Z4, Canada

Connor M. Kerns

Department of Applied Disabilities Studies, Brock University, 1812 Sir Issac Brock Way, St. Catherines, ON, L2S 3A1, Canada

Kendra Thomson

Azrieli Adult Neurodevelopmental Centre at the Centre for Addiction and Mental Health, 1451 Queen Street West, Toronto, ON, M63 1A1, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

VL and JW conceptualized the study. VL, NV, FR, PTM, AP, and TS worked together under the supervision of JW to collect the data. VL and JW analyzed and interpreted the data. VL wrote the manuscript. NV, FR, PTM, AP, TS, AI, SHA, AE, KW, CMK, MCL, JL, KT, and JW provided edits and revisions. All authors read and approved the final manuscript before submission.

Corresponding author

Correspondence to Vivian Lee .

Ethics declarations

Ethics approval and consent to participate.

The study was approved by the research ethics board at the researchers’ institution (York University and Carleton University), an academic hospital (Ron Joyce Children’s Health Centre), and by the research review committee at two community-based agencies (Kerry’s Place Autism Services and Kinark Autism Services). All methods in the study were performed in accordance with the Declaration of Helsinki. Participant provided electronic informed consent to participate in the study.

Consent for publication

No individual’s personal data (details, images, or videos) are included in this manuscript.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Lee, V., Vashi, N., Roudbarani, F. et al. Utility of a virtual small group cognitive behaviour program for autistic children during the pandemic: evidence from a community-based implementation study. BMC Health Serv Res 24 , 685 (2024). https://doi.org/10.1186/s12913-024-11033-9

Download citation

Received : 27 July 2023

Accepted : 23 April 2024

Published : 30 May 2024

DOI : https://doi.org/10.1186/s12913-024-11033-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Virutal CBT
• Emotional regulation
• Social skills
• Community services

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]