Real-Time Monitoring of Vaccination Campaign Performance Using Mobile Phones — Nepal, 2016
Weekly / October 7, 2016 / 65(39);1072–1076
David H. Oh1; Alya Dabbagh, PhD1; James L. Goodson, MPH2; Peter M. Strebel, MBChB1; Sanjita Thapa, MPH3; Jagat Narain Giri, MPH3; Sagar Ratna Shakya, MD3; Sudhir Khanal, MPH4 (View author affiliations)View suggested citation
What is already known about this topic?
Rapid convenience monitoring (RCM) has been used for more than 20 years as the primary method for monitoring mass vaccination campaigns. Its effectiveness and contribution to increasing campaign quality has been documented previously. Currently, RCM is implemented using paper reporting systems; however, advancements in information and communications technology make it possible to conduct RCM using mobile phones (RCM-MP).
What is added by this report?
In February 2016, RCM-MP was pilot tested during a measles-rubella vaccination campaign in Nepal. The application of this technology resulted in 87% timeliness and 94% completeness of monitoring data reporting and found that, although 95% of children were vaccinated, 42% of areas required corrective vaccination activities. More than 90% of monitors and all district and national supervisors who responded to the survey recommended its future use. Challenges faced by this method included connecting to mobile networks, small phone screen size, and capturing Global Positioning System coordinates.
What are the implications for public health practice?
Achieving measles and rubella elimination worldwide will be an important milestone in public health, and every effort toward elimination, including vaccination campaigns, should be of high quality and improved with innovations. One way for improving the quality of vaccination campaigns is to optimize the use of mobile phones for monitoring campaign implementation, with faster data transmission, analysis, decision-making, and increased accountability among levels of the health system. While taking into account costs, existing infrastructure, and the availability of resources, the program implemented in Nepal might be used as a model for other countries.
In 2012, the Global Vaccine Action Plan* established a goal to achieve measles and rubella elimination in five of the six World Health Organization (WHO) regions (194 countries) by 2020 (1). Measles elimination strategies aim to achieve ≥95% coverage with 2 routine doses of measles-containing vaccine (2), and implement supplementary immunization activities (SIAs)† in settings where routine coverage is low or where there are subpopulations at high risk. To ensure SIA quality and to achieve ≥95% SIA coverage nationally, rapid convenience monitoring (RCM) is used during or immediately after SIAs (3,4). The objective of RCM is to find unvaccinated children and to identify reasons for nonvaccination in areas with persons at high risk, to enable immediate implementation of corrective actions (e.g., reassigning teams to poorly vaccinated areas, modifying the timing of vaccination, or conducting mop-up vaccination activities). This report describes pilot testing of RCM using mobile phones (RCM-MP) during the second phase of an SIA in Nepal in 2016. Use of RCM-MP resulted in 87% timeliness and 94% completeness of data reporting and found that, although 95% of children were vaccinated, 42% of areas required corrective vaccination activities. RCM-MP challenges included connecting to mobile networks, small phone screen size, and capturing Global Positioning System (GPS) coordinates. Nonetheless, use of RCM-MP led to faster data transmission, analysis, and decision-making and to increased accountability among levels of the health system.
As part of Nepal’s continuing progress toward measles elimination (5), a nationwide measles-rubella vaccination campaign directed at children aged 9–59 months was implemented in four phases§ during September 2015–March 2016 in all 75 districts of Nepal, which are administratively divided into village development committees (VDCs) and further divided into VDC wards. Intra-campaign monitoring was implemented according to the WHO Comprehensive Field Guide for Planning and Implementing High Quality Supplementary Immunization Activities for Measles and Rubella and other Injectable Vaccines (6). In each VDC, the SIA was conducted over 10–15 days, and RCM was conducted on the third and seventh days in VDCs identified by the Ministry of Health (MoH) as high risk (i.e., <90% immunization service delivery coverage; location near the India border with population movement; large population; and hard-to-reach).
All 33 districts included in the second phase of the campaign during February 2016 used paper-based RCM, but the MoH and WHO-Nepal selected 10 districts among them that included a mix of high- and low-performance in immunization service delivery and different geographic topographies (five were in the plains and five were hilly) for pilot testing RCM-MP on a limited scale. Thus, in the 10 pilot districts, there was a mix of VDCs where RCM was conducted using paper forms or mobile phones. For the RCM-MP, data collection forms were programmed into an electronic data collection tool¶ and loaded onto Android phones.** National-level staff members were trained on use of the phones and software 2 weeks before deployment, and the national staff team then provided training for field monitors. Electronic data visualization software†† was used to create two dashboards connected directly to the server, where data were uploaded, enabling real-time data visualization of SIA implementation performance indicators on the dashboards for national and district supervisors (Table 1). The first dashboard was designed to be action-oriented, displaying overall SIA performance, reasons for nonvaccination and refusal, and monitoring results by VDC and date of visit. The second dashboard was created to track monitors’ activities using a map showing GPS coordinates collected at each household and to display the number of reports received and results found by district (Figure).
In each pilot district, 10 monitors using mobile phones were responsible for two VDCs each. In each VDC, one or two visits were made to complete one in-house and one out-of-house RCM form§§ during each visit. In total, 100 monitors collected RCM-MP data on 11,093 children in 377 visits in 196 VDCs. Among monitored children, 10,583 (95%) were vaccinated; 159 (42%) of 377 visited areas required remedial action (Table 2). Among the 311 incompletely vaccinated households, the primary reasons for nonvaccination were child absence during an SIA (126 of 311 [41%]) or vaccine refusal (68 of 311 [22%]); the primary reason for vaccine refusal was child being sick (53 of 68 [78%]). Six months after completion of the SIA, no RCM reports had been received at the central level from districts using paper-based RCM. In contrast, 94% (377 of 400) of expected reports were received from 98% (196 of 200) of VDCs where RCM-MP was conducted, and 87% (328 of 377) of these reports were received on the same day the data were collected.
All 100 monitors who used mobile phones, 10 district supervisors, and six national supervisors were asked to respond to a questionnaire about their experience. Completed questionnaires were received from 93 monitors and three district and five national supervisors. Common challenges reported by the monitors were finding and connecting to a third generation (3G) or Wi-Fi network¶¶ (56 of 93 [60%]) and mistakenly striking incorrect keys (21 of 93 [22%]). Recording GPS location also was a challenge; only 61% (5,730 of 9,425) of expected GPS coordinates were captured. Despite challenges, 64% (54 of 84) of monitors found the mobile technology easy or somewhat easy to use, and 51% (47 of 93) found it easier than paper data collection.*** Ninety-six percent (78 of 81) of monitors thought that the technology increased data accuracy, and 90% (71 of 79) recommended its future use. Among the three district and five national supervisors, all found the technology helpful or somewhat helpful, and seven of the eight looked at the dashboard “almost every day” or “every few days.” Of the three district supervisors, two reported using the reasons given for nonvaccination to design specific actions, and all three took action in all VDCs requiring remedial action. All five national supervisors reported that the most helpful feature was having data at the central level in a timely manner, and four supervisors reported that tracking monitors’ activities through GPS coordinates and automatic analytics in real-time on the dashboard also was useful (Figure). All eight district and national supervisors recommended its future use in campaign settings.
Although the reported SIA administrative coverage was >100%††† for the 10 districts included in the pilot study (Table 2), RCM-MP identified a total of 510 (5%) unvaccinated children and 159 (42%) visited areas that needed mop-up vaccination activities. Many unvaccinated children were identified in four districts that reported >100% administrative coverage. The majority of monitors found the mobile technology easy or somewhat easy to use, and about half found it easier to use than paper-based RCM. More than 90% of monitors and all district and national supervisors who completed surveys recommended its future use.
Use of RCM-MP in Nepal resulted in increased reporting timeliness and completeness. The automated calculations and analyses displayed on dashboards eliminated the potential for manual calculation errors, a previous problem with paper-based RCM data (4). Mobile data collection provided information more rapidly to higher administrative levels than did paper-based RCM; most RCM results were available on the same day as monitoring. National supervisors cited the rapid availability of data as the most helpful aspect of this technology for ensuring SIA quality.
Timely reporting resulted in better supervision from the national and subnational levels to the VDCs. Aggregated reasons for nonvaccination and refusal were used by supervisors to tailor vaccination strategies and to take immediate actions, which had not been possible with paper-based RCM. However, supervisors pointed out that there was no RCM mechanism for reporting actions taken in the VDCs with unvaccinated children. Therefore, even if supervisors did take action to improve SIA coverage, the status of VDCs initially marked as needing action did not get updated. Future RCM implementation can address this deficiency by including a reporting system for actions taken in poorly performing areas; this change will enable supervisors to monitor follow-up actions routinely and ensure accountability of vaccination teams.
The findings in this report are subject to at least two limitations. First, the selection of VDCs for RCM was purposeful, and convenience sampling was used within VDCs; therefore, RCM results were not representative of the population and should not be considered as coverage estimates. Second, the impact of RCM-MP on the goal of achieving ≥95% SIA coverage was not quantitatively assessed because of time and resource limitations.
In the RCM-MP pilot, the main challenge to submitting real-time data was difficulty connecting to 3G or Wi-Fi networks. To avoid reliance on 3G coverage, encoding RCM data into text messages is a potential option. However, with expansion of 3G networks of better quality and coverage, it is anticipated that this problem will be overcome. As countries continue to implement campaigns to eliminate measles, rubella, and other vaccine-preventable diseases, the use of mobile phone technology for campaign monitoring might be considered to improve information systems and, ultimately, the quality of campaigns.
Bill and Melinda Gates Foundation; Remi Valvik, Jørn Klungsøyr, Zegeba; Ramon Martinez, Pan American Health Organization.
Corresponding author: Alya Dabbagh, firstname.lastname@example.org, +41 22 7914693.
1Department of Immunization, Vaccines, and Biologicals, World Health Organization; 2Global Immunization Division, Center for Global Health, CDC; 3Immunization Preventable Disease, Nepal Country Office, World Health Organization; 4Immunization and Vaccine Development, World Health Organization South-East Asia Regional Office, Delhi, India,
- Perry RT, Murray JS, Gacic-Dobo M, et al. Progress toward regional measles elimination—worldwide, 2000–2014. MMWR Morb Mortal Wkly Rep 2015;64:1246–51. CrossRefexternal icon
- World Health Organization. Measles vaccines: WHO position paper. Wkly Epidemiol Rec 2009;84:349–60.
- Luman ET, Cairns KL, Perry R, Dietz V, Gittelman D. Use and abuse of rapid monitoring to assess coverage during mass vaccination campaigns. Bull World Health Organ 2007;85:651. CrossRefexternal icon
- Teixeira AM, Samad SA, Souza MA, Segatto TC, Morice A, Flannery B. Brazilian experience with rapid monitoring of vaccination coverage during a national rubella elimination campaign. Rev Panam Salud Publica 2011;30:7–14.
- Khanal S, Sedai TR, Choudary GR, et al. Progress toward measles elimination—Nepal, 2007–2014. MMWR Morb Mortal Wkly Rep 2016;65:206–10. CrossRefexternal icon
- World Health Organization. Planning and implementing high quality supplementary immunization activities for measles and rubella and other injectable vaccines: a comprehensive field guide. http://www.who.int/immunization/diseases/measles/SIA-Field-Guide_DRAFT.pdfpdf iconexternal icon
† SIAs generally are carried out using two approaches. An initial, nationwide catch-up SIA targets all children aged 9 months to 14 years; it has the goal of eliminating susceptibility to measles in the general population. Periodic follow-up SIAs then target all children born since the last SIA. Follow-up SIAs are generally conducted nationwide every 2–4 years and usually target children aged 9–59 months; their goal is to eliminate any measles susceptibility that has developed in recent birth cohorts and to protect children who did not respond to the first measles vaccination. The exact age range for follow-up SIAs depends on the age-specific incidence of measles, coverage with 1 dose of measles-containing vaccine, and the time since the last SIA.
§ Each phase targeted a different set of districts. RCM-MP was implemented during the second phase, in February 2016.
** Low-cost Android phones with adequate functionalities for running the software and transmitting data were used. All phones were provided by the national program. Because of logistical constraints, monitors were not encouraged to use personal phones.
†† Tableau software was used to visualize the incoming data in real-time (http://www.tableau.com/external icon).
¶¶ Wi-Fi provides Internet connection for nearby electronic devices, whereas 3G connections can be made anywhere with service coverage for the specific telecommunications network being utilized (https://www.itu.int/osg/spu/ni/3G/technology/index.htmlexternal icon).
*** Denominators varied according to the number of responses. Not all monitors responded to all questions, and monitors also could select more than one response for certain questions.
††† Administrative coverage is the total number of doses given to the target population, divided by the estimated target population. Values >100% indicate that the intervention reached more persons than the estimated target population.
TABLE 1. Rapid convenience monitoring (RCM) indicators visualized in real-time on dashboards* used by national and subnational supervisors for monitoring a measles and rubella vaccination campaign in 10 districts — Nepal, 2016
|Action||Percentage of children vaccinated, aggregated||100 x children vaccinated/children assessed||Checks performance aggregated over the entire district, region, or country to detect widely underperforming areas and assess overall performance|
|Reasons cited for nonvaccination||Frequency of each nonvaccination reason cited||Allows supervisors to use the most frequently cited reasons for tailoring which type of action to take in each community or throughout the district or region|
|Reasons cited for refusal||Frequency of each refusal reason cited||Allows supervisors to use the most frequently cited reasons for tailoring how to address refusal in specific communities|
|Action trigger||“No Action” if all in-house and out-of-house criteria are met
“Action” if any of the criteria failed
|Automatically calculates and highlights which communities require additional vaccination activities|
|Monitoring||RCM geographic coverage and clusters of missed households||Plots the Global Positioning System coordinates of all households monitored on a map, color-coded by whether or not they are completely or incompletely vaccinated||Shows where monitoring was done, and areas where monitors may have missed, as well as clusters of nonvaccinated households|
|RCM reporting completeness||Number of communities reporting and number of reports received per community||Checks whether or not communities are reporting, and whether or not they are submitting the expected number of reports|
TABLE 2. Administrative data* and rapid convenience monitoring (RCM) measles and rubella vaccination campaign data for 10 pilot districts — Nepal, 2016
|District||Administrative data||RCM data|
|No. in target population||Total vaccinated
|Total no. of children monitored with mobile phones||Total children vaccinated
|RCM: action-triggered visited areas
|Baitadi||27,324||27,434 (100.4)||1,098||1,060 (96.5)||12 (33.3)|
|Banke||57,244||56,008 (97.8)||1,171||1,064 (90.9)||25 (64.1)|
|Bardiya||39,487||37,388 (94.7)||882||832 (94.3)||11 (36.7)|
|Dang||52,505||61,669 (117.5)||1,111||1,039 (93.5)||15 (37.5)|
|Kanchanpur||57,876||55,290 (95.5)||1,109||1,063 (95.9)||21 (56.8)|
|Kaski||41,584||41,088 (98.8)||1,079||1,005 (93.1)||21 (53.8)|
|Lamjung||15,604||14,634 (93.8)||1,161||1,126 (97.0)||17 (40.5)|
|Nawalparasi||59,745||61,670 (103.2)||1,174||1,143 (97.4)||13 (32.5)|
|Rupandehi||108,611||109,799 (101.1)||1,140||1,108 (97.2)||13 (38.2)|
|Surkhet||41,598||39,719 (95.5)||1,168||1,143 (97.9)||11 (27.5)|
|Total||501,578||504,699 (100.6)||11,093||10,583 (95.4)||159 (42.2)|
FIGURE. Example outputs of real-time online monitoring dashboards used by national and subnational supervisors in 10 districts during a measles-rubella vaccination campaign — Nepal, 2016*
* Top panel shows a portion of the action-oriented dashboard with individual rapid convenience monitoring results. Bottom panel shows Global Positioning System (GPS) coordinates and vaccination status of households monitored. Action dashboard, Nepal: http://ais.paho.org/phip/viz/who_im_nepal5.aspexternal icon. Monitoring dashboard, Nepal: http://ais.paho.org/phip/viz/who_im_nepal6.aspexternal icon.
Suggested citation for this article: Oh DH, Dabbagh A, Goodson JL, et al. Real-Time Monitoring of Vaccination Campaign Performance Using Mobile Phones — Nepal, 2016. MMWR Morb Mortal Wkly Rep 2016;65:1072–1076. DOI: http://dx.doi.org/10.15585/mmwr.mm6539a5external icon.
MMWR and Morbidity and Mortality Weekly Report are service marks of the U.S. Department of Health and Human Services.
Use of trade names and commercial sources is for identification only and does not imply endorsement by the U.S. Department of Health and Human Services.
References to non-CDC sites on the Internet are provided as a service to MMWR readers and do not constitute or imply endorsement of these organizations or their programs by CDC or the U.S. Department of Health and Human Services. CDC is not responsible for the content of pages found at these sites. URL addresses listed in MMWR were current as of the date of publication.
All HTML versions of MMWR articles are generated from final proofs through an automated process. This conversion might result in character translation or format errors in the HTML version. Users are referred to the electronic PDF version (https://www.cdc.gov/mmwr) and/or the original MMWR paper copy for printable versions of official text, figures, and tables.
Questions or messages regarding errors in formatting should be addressed to email@example.com.