Quick Downloads
Based on research we have conducted on digital health technologies in lower- and middle-income countries, we identified the following set of lessons learned from the implementation of tools that were either developed for or adapted to the context of the COVID-19 pandemic.
Introduction
This cross-case synthesis presents high-level lessons and challenges from our research on the implementation of digital tools in various countries, which can be explored further in detailed case studies. These findings are relevant for policy makers, funders, and nongovernmental leaders seeking to understand how implementers and governments have introduced, adapted, and scaled digital tools, especially in the context of COVID-19.
In this synthesis we highlight the most critical strengths and challenges from the implementation case studies. The hyperlinks on this page take the reader to the corresponding background and illustrative examples that support our findings within each case study.
Summary of Strengths and Challenges
Using the mHealth Assessment and Planning for Scale Toolkit, we assessed the performance of implementations of digital tools across six domains: groundwork, partnerships, financial health, technology and architecture, operations, and monitoring and evaluation. Three core lessons emerged from our findings:
- User-centered design is key to the widespread adoption of digital tools. Each development and implementation team identified an ongoing practical problem and established a clear, user-friendly solution—whether those users were health workers or members of the public. To ensure continued ease of use, Burkina Faso’s implementation of CommCare prioritizes offline functionality and access to multimedia resources for users with low connectivity and digital literacy. The project also includes a robust troubleshooting hierarchy and an iterative approach; users feel connected to the process when they see that their feedback has been incorporated into newer versions of the app. CommCare’s modular “no code” platform has enabled the MOH and Terre des hommes to independently maintain and modify the scope of the app to address emerging health priorities. In Uttar Pradesh, India, the Unified COVID-19 Data Platform was designed to be an end-to-end surveillance platform that meets the needs of users across the continuum of care. The platform’s modular design is able to accommodate different use cases and stakeholders from tracking teams to testing laboratories to patients accessing test results or quarantining at home.
- Strong, country-led partnerships are essential for scaling up digital tools successfully. The solutions that scaled up most successfully during the COVID-19 pandemic benefited from close, long-established partnerships with the government to establish and promote the tools. SORMAS in Nigeria was initially co-developed by the Nigeria Centre for Disease Control and the Helmholtz Center for Infection Research in 2014, and the use of District Health Information Software 2 (DHIS2) in Sri Lanka reflects a partnership of more than a decade between the Ministry of Health and the Health Information Systems Programme. The tools we assessed in Vietnam were catalyzed by demand from the highest levels of government. In India, Prime Minister Narendra Modi launched the COVID Vaccine Intelligence Network (CoWIN), an end-to-end solution that captures each step of the vaccination process from appointment scheduling to certification. India’s COVID-19 vaccination program was coordinated by several government ministries and private partners who were committed to scaling CoWIN which helped administer 1 billion doses within a span of nine months.
- Using adaptable digital tools enables implementers to focus on the content rather than the technology. Within the global digital health community, there has been a push toward the use and development of global goods, or digital health tools that are adaptable and designed to be used in many contexts. This adaptability enables implementers to focus on user-centered design and scale. Each of the digital tools we assessed built on existing tools already in use, in-country or elsewhere—with the exception of tools assessed in India and Vietnam. For example, Living Goods in Uganda partnered with Medic in part due to its experience in over ten other countries. In the context of crisis response, the adaptability of the tools to quickly aid in response has been especially critical. Praekelt worked with the South African National Department of Health to deploy a COVID-19 helpline in March 2020, and the same technology was adapted for other specific use cases, like higher education, within weeks.
Common challenges in implementation of the digital tools also emerged. Few of the tools in our assessment have established sustainable financing, business models or methods to regularly and rigorously evaluate their impact. Monitoring and evaluation are not typically prioritized during health emergencies, but the long-term sustainability of these tools remains at risk when funding is not guaranteed and when evidence of improved health outcomes is limited.
Overview of Implementations Profiled
In our assessment, we examined the implementation of digital tools with the goal of identifying common success factors for digital health programs that have achieved scale in-country. The following case studies provide a set of examples across diverse geographies, use cases, and technologies.
| Country Profiled | Implementation Story | Digital Tool(s) | Use Case(s) | Indicators of Scale | Primary Funding |
|---|---|---|---|---|---|
|
Burkina Faso |
Scale of an existing app enabled rapid adaptation for COVID-19 | CommCare | Case management, training and risk communication |
CommCare has scaled to 80 percent of primary care facilities in Burkina Faso and is used by over 6,300 health care workers |
Donor funding |
|
India |
The digital backbone for the COVID-19 vaccination program in India | CoWIN |
Vaccine supply chain, registration and certification |
By January 2022, the platform had over 940 million registered users; CoWIN helped administer 1 billion doses over nine months |
Government and donor funding |
| Uttar Pradesh, India |
An integrated end-to-end surveillance platform for COVID-19 response |
Unified COVID-19 Data Platform |
Surveillance, contact tracing, facility & lab management |
As of July 2021, the platform enabled the State of Uttar Pradesh to track nearly 49.6 million potential cases across the continuum of care and manage 1.7 million positive cases. |
Government and donor funding |
| Nigeria |
Adapting a fully integrated surveillance system to track COVID-19 |
SORMAS |
Routine surveillance |
SORMAS has been rolled out to all 36 states and the Federal Capital Territory; it has also rolled out in countries including Ghana, Germany and Fiji |
Donor funding, transitioning to country sources |
| South Africa |
A chatbot tool for pandemic response |
HealthConnect |
Risk communication & community engagement |
One of the HealthConnect tools, HealthCheck, has supported over 10 million higher education campus screenings |
Donor funding, transitioning to country sources |
| Sri Lanka |
Early action to track and prevent COVID-19 |
DHIS2 COVID-19 package | Health management information system |
The Port of Entry module was developed and deployed within a few days of the first reported cluster of COVID-19 cases |
Multiple funding sources for COVID-19 modules, including the government and implementer |
| Uganda |
A mobile app to support community health workers in pandemic response and primary care |
Smart Health |
Case management |
Over 7,800 community health workers use the Smart Health app in Uganda, and referrals and treatments for common conditions (e.g. for childhood pneumonia) increased from 2019 to 2020 |
Donor funding with private sector support |
| Vietnam |
Government launches digital health apps to contain COVID-19 |
NCOVI, Bluezone, others | Contact tracing, risk communication, & community engagement |
As of March 2021, there were over 30 million downloads of the Bluezone app. By August 2020, NCOVI had over 7.5 million downloads |
Government funding with private sector support |
Lessons by Domain
From the outset, all implementation partners, including governments, should have a clear understanding of their vision for the digital tool, including its impact and operation in the local context.
Define the use case and value of the tool
What is the problem this tool will solve, and for whom? Clear answers to these questions are critical to obtaining buy-in from potential partners and funding for the tool. All of the tools in our assessment had a clearly articulated vision. Even the client-focused tools often addressed the needs of governments and the broader health ecosystem.
- In Uganda, the Smart Health app developers aimed for usability and simplicity in design, especially when adding new functionality, keeping in mind the Community Health Workers with the lowest technological literacy. E-learning modules were developed on separate platforms to maintain usability and refresh the knowledge of users.
- In Uttar Pradesh, India, officials worked closely with the Uttar Pradesh Technical Support Unit team to conceptualize and develop what would become its Unified COVID-19 Data Platform: a comprehensive, integrated digital tool for all stakeholders to track and manage the state’s COVID-19 response. It was the first digital tool in India that covered the entire continuum of care, and in less than two months, the platform evolved into a modular, end-to-end solution for case reporting and management, contact tracing, workflow integration, data aggregation, stakeholder engagement, and strategic planning.
Design a human-centered solution aligned with user priorities and use pilots to test the solution
Successful digital tools effectively address user needs in their design. Pilot testing has ensured that these tools work from a technical perspective and are user-friendly for target audiences.
- As partners in Nigeria worked together to develop SORMAS, they addressed the challenges faced by the country in its Ebola response by interviewing stakeholders across all levels of the health system and running multiple pilots throughout the tool’s development.
- The Vietnamese government worked closely with developers to implement several tools that addressed stakeholder feedback. Some of the first location-tracking tools raised concerns about data privacy and mass surveillance, so they developed an additional tool, Bluezone, to support contact tracing using Bluetooth instead.
Understand the technology’s enabling environment
Planning for a program’s future involves deeply understanding the local context and proactively identifying roadblocks that might hamper progress.
- The Ministry of Health in Sri Lanka launched a master’s degree program in health informatics over a decade ago to build a foundation of expertise in using the national health data systems. This investment enabled rapid action early in the COVID-19 pandemic.
- After working in South Africa through the MomConnect program for nearly seven years, Praekelt had the experience to easily navigate local challenges and collaborate with longtime government partners to address challenges proactively. The COVIDConnect platform is available via WhatsApp and USSD (a type of text message) for those without data plans, and partnerships with telecom providers ensure free access to the messages.
- In India, CoWIN leveraged an existing public, national-scale digital platform, the Electronic Vaccine Intelligence Network or eVIN, as its base. eVIN’s modular, interoperable design had been developed to improve India’s vaccine supply chain management in 2015 and was easily adapted for a new use case in the context of COVID-19.
Developing and maintaining partnerships between government stakeholders, implementing partners, and tool developers is necessary to successfully expand the reach of a tool, particularly given the complex and interconnected nature of health systems.
Establish relationships and credibility through a foundational partnership, particularly a government agency
Starting with one or two foundational partnerships and expanding from there is one method that implementers have used to successfully scale up their digital tools. All of the tools we assessed rely on government partners to expand their reach. By working with organizations that already have a presence in-country, developers can draw on their expertise and relationships with others.
- In Burkina Faso’s launch of CommCare, the implementing team (led by the nongovernmental organization Terre des Hommes) worked closely with the Ministry of Health for more than a decade to create the tool. As a result, it has been fully integrated with the country’s health infrastructure and transmits data for key indicators automatically to the government’s health information system. The tool is considered a national priority in the country’s digital health strategy.
- NCOVI and Bluezone are the result of strong multisectoral partnerships between the Vietnamese Ministry of Health, the Ministry of Information and Communications, telecommunications groups, software developers, and medical experts.
- Collaboration between public and private stakeholders drove quick statewide development and adoption of the Unified COVID-19 Data Platform in Uttar Pradesh, India. The Department of Health and Family Welfare and the Department of Medical Education and Training coordinated the state’s COVID-19 response together, and the medium- to long-term thinking and support from technical support partners enabled the quick development, implementation, adaptation, and rollout of the digital platform.
Ensuring sustainable financing for digital health tools in contexts with constrained resources is an ongoing challenge. Many projects are reliant on donor funding; however, a model with predictable revenue sources or explicitly allocated government funding is more likely to be sustained. Most of the tools we examined have not yet achieved a long-term funding model.
Develop business models beyond donor investment by demonstrating value
Digital tools that are able to monetize and charge for services can establish a funding stream from users—either clients or their insurance providers—rather than relying on donors. This lesson applies primarily outside of a crisis situation such as the COVID-19 pandemic when facilitating easy access is the priority.
- While the tools we profiled in the case studies are used during COVID-19, we did find examples of two private sector tools during the broader landscape assessment that have strong business models. CarePay and mDoc established models in which users (providers and their clients) pay for services. CarePay generates revenue through management fees from donors funding health care and a per-person fee model for corporate and public insurance clients, while mDoc receives payments from health maintenance organizations, employers (from their business-facing operations), and clients (from their consumer-facing subscription model).
Example of how the M-TIBA implementation of CarePay in Kenya provides benefits for multiple user types
Digitize and standardize processes to establish a predictable value
Even for tools without established revenue streams, digital health program implementers can work to lower costs and standardize their offering to better calculate the value it provides to the health system and demonstrate financial value.
- Living Goods and Medic have worked to automate and standardize task and decision support checklists for community health workers (CHWs) in Uganda. As part of this effort, they are shifting the workflow from reactive to proactive outreach — in part by using predictive modeling — to improve the impact of the CHW program. Rather than relying on calls from clients, they can better identify people and communities that are more likely to require care. In this way, they improve the efficiency and cost benefits of the program.
- In Burkina Faso, CommCare is used to simplify complex clinical protocols for health workers. Terre des hommes has worked with partners to leverage artificial intelligence and machine learning to make data processing more efficient, improve measurements, and provide health care workers with real-time recommendations based on their performance history, as well as generate smart dashboards and predictive models for epidemiological surveillance. This saves valuable time and enables workers and supervisors to increase their reach.
The usability, integration, and adaptability of the tool itself are critical to the success of any digital health program. While digital tools are only part of a full e-health system, they must adapt and expand with the needs of the system to have an impact.
Develop a reliable product that can improve and adapt over time
Digital health tools must have the capacity to adapt to shifting priorities, as highlighted recently during the COVID-19 pandemic. All of the tools in our assessment had to adapt their technology during the pandemic, although some faced more challenges than others.
- SORMAS in Nigeria was designed specifically to be modular and adjust as new pathogens emerge. As a result, the COVID-19 module was ready by January 2020 and integrated seamlessly with the existing technology.
- Praekelt developed a new technology platform, turn.io, after its experience adapting the MomConnect program to contexts outside of South Africa. Because the messaging technology is adaptable and readily scaled, the tool was adapted for multiple scenarios in South Africa during the early months of the pandemic.
- In India, CoWIN’s phased launch was particularly useful in identifying both the technical limitations of the CoWIN platform and the programmatic limitations of the vaccination drive leading up to the official launch. In response to technical issues, and duplications due to data entry errors, CoWIN’s adaptable design enabled the technology team to quickly build new, key features that enabled users to search pin codes and choose their vaccination centers. Furthermore, the government invited third-party application developers to integrate their APIs with the CoWIN platform, enabling these applications to provide a variety of value-added services such as booking appointments.
Work with interoperable models and systems to enable configurability
For digital tools to become part of the broad health ecosystem, they need to work with other data systems and technologies.
- “Intuitive, Modular, and Integrated” were the three design principles behind the Unified COVID-19 Data Platform. This modular, workflow-based system enabled the platform to grow and adapt to meet emerging needs for multiple stakeholder groups. For instance, health workers can refer patients to facilities, officials can leverage the Decision-Making Dashboard to perform dynamic modelling, and citizens can access their COVID-19 test results.
Key Design Principles for the Unified COVID-19 Data Platform
- Sri Lanka used DHIS2 to manage multiple aspects of the country’s COVID-19 response, including port-of-entry tracking and contact tracing. Because the underlying DHIS2 platform had existing data infrastructure with known flexibility, it could integrate with other systems like those used by immigration officials. DHIS’s interoperability is one reason for its use in more than 70 lower- and middle-income countries.
- Smart Health gives the government access to real-time data dashboards that are standardized and integrated with DHIS2. Living Goods has provided real-time data on health outcomes for the Kenyan government since 2019 and is currently working on similar integration in Uganda.
Interoperability of DHIS2 across sectors as part of Sri Lanka’s response to COVID-19
All of the digital tools we assessed require significant maintenance and support, and therefore planning for ongoing human resource needs (including training and staffing) is necessary to ensure a tool or program is sustainable.
Invest in regional expertise
In many places, digital health programs are implemented to replace or augment existing workflows (for example, by using electronic forms in place of paper reports). Strengthening local capacity and conducting community outreach helps make this transition to digital tools successful, and may increase buy-in for the tools.
- The HigherHealth team in South Africa invested heavily in outreach to campus leadership and students, because they recognized that working toward a digital screening and passport system required strong community support. The team set up a peer network of thousands of students, which resulted in true brand recognition and staying power for the HealthCheck app.
- In Sri Lanka, there was a large focus on local developer expertise. The connection between the Health Information Systems Programme, the DHIS2 core team and the global community of DHIS2 experts helped local developers get valuable feedback on their work and made it possible for other countries to benefit from their pioneering innovations.
Invest in capacity to use the solution locally
The sustainability of a tool depends on local capacity and usability in the country context. This, in turn, requires additional forms of user support such as language accessibility, training, and user guides.
- The partnership that led to the development of SORMAS in Nigeria has resulted in full government ownership of the tool. The program is run by a task force at the Nigeria Centre for Disease Control, with the developers and donors providing technical and financial support. The team runs trainings for multiple user types in several languages and is working toward a cascade training approach, in which one group in turn provides training to another group.
- In Vietnam, the more people who download the NCOVI and Bluezone apps, the more effective they are. Promotional messaging in multiple languages has been about “challenging the virus with the strength of our community,” and the MOH and MIC encouraged all smartphone users to install Bluezone for themselves and three others: “Protect yourself, protect the community.” The message “Let’s install Bluezone” also appeared on people’s smartphones next to the phone carrier’s logo.
As a digital health tool expands in scale of use over time, tracking the progress and efficacy of its implementation helps identify challenges and demonstrate impact.
Measure the tool’s impact and maintain an adaptable approach to implementation
- The Integrated e-Diagnostic Approach in Burkina Faso has undergone multiple independent evaluations which suggest improved health outcomes and quality of care. It has been estimated that the tool saves between US$830,000 and US$1.7 million per year from reduced training times for CHWs and reduced paper consumption.
- In Uganda, Living Goods invested in a randomized controlled trial to evaluate the impact of its model on maternal and child mortality in Uganda, and 2014 results showed a 27 percent reduction in under-five mortality after three years, at an estimated cost of US$68 per life saved.
Conclusion
Though the COVID-19 pandemic has been a catalytic opportunity for digital health, scaling and sustaining digital innovations remains a challenge. This cross-case synthesis builds on existing evidence and presents lessons from our research on the implementation of digital tools that were developed for or adapted to the context of the COVID-19 pandemic. These findings are relevant for policy makers, funders, and nongovernmental leaders seeking to understand best practices for the implementation of digital tools. Measuring the impact of programs has been particularly challenging throughout the pandemic, but it is clear that digital health can increase access to health care, increase the quality of care delivered, diminish costs of providing care, and empower patients to manage their own health.
The hyperlinks on this page take the reader to the corresponding background and illustrative examples that support our findings within each case study.