Tool for Assessing Asthma Referral Systems

The Tool for Assessing Asthma Referral Systems (TAARS) was developed by evaluators in the Centers for Disease Control and Prevention (CDC) National Asthma Control Program. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the CDC.
Recommended citation: Tool for Assessing Asthma Referral Systems (TAARS). Atlanta, GA: Centers for Disease Control and Prevention, National Center for Environmental Health, Division of Environmental Health and Health Effects, Air Pollution and Respiratory Health Branch, October 2017.

Preface

The Tool for Assessing Asthma Referral Systems (TAARS) is intended for use by asthma control programs as a guide in helping to understand the how effectively their referral systems are operating within their programs. Programs can use TAARS to conduct a comprehensive or more focused assessment of their referral system. Assessment may include determining how a referral system is structured, how networks are formed, whether appropriate written referral protocols and guidelines exist, the processes providers follow to refer clients, how well referrals are tracked and followed up, and barriers to referral initiation and referral completion. Using TAARS can also help you better understand how the different types of referrals among the different service providers operate in communities.

TAARS: Tool for Assessing Asthma Referral Systems

TAARS was adapted from the Referral Systems Assessment and Monitoring Toolkit, originally developed by MEASURE Evaluation for the President’s Emergency Plan for AIDS Relief (PEPFAR) for HIV/AIDS service providers. This tool can be used as is or it can be adapted to the specific design and purpose of your particular asthma referral system. It may be most useful in the context of overall evaluation planning and implementation, as recommended in the Learning and Growing through Evaluation series (https://www.cdc.gov/asthma/program_eval/guide.htm). Specifically, TAARS can help guide the focus and methodological considerations of a broader evaluation, such as is laid out in Steps 3 and 4 (Focus the Evaluation Design and Gather Credible Evidence) of the CDC’s Framework for Program Evaluation (https://www.cdc.gov/asthma/program_eval/guide.htm).¹

Background

Integrating health services across clinical and community-based settings improves the efficiency of the health system and provides greater access to services for clients, ultimately improving the well-being of clients. Providing comprehensive asthma control services entails integration and linkages between clinical services and services that exist outside of the health system, such as social services, educational services, and community-based services. For this reason, referral systems, which may also be called linkage systems, should be efficient, effectively connect people to needed services, and provide sufficient data to assure optimal care management.

Community services sites often use their own unique, sometimes entirely unstructured, referral networks. These networks may rely solely on informal, personal recommendations. In these settings, there is not just one referral system but several unstructured, very fluid systems operating simultaneously. A tool like TAARS can help you better understand the nuances of those systems and help identify where the gaps are and where improvements can be made. TAARS can also be useful in developing contracts or agreements with partners, describing the processes each uses for making or receiving referrals to or from programs, and to potentially encourage some level of standardization or consistency across programs. Further, and importantly, documentation of the context for the referral system will promote understanding and communication across programs and serve as institutional memory, especially valuable when staff turnover occurs.

¹ For additional guidance on evaluating referrals related to asthma home visiting programs. Please see Appendix E of Evaluation of Services and Health Systems Interventions, Learning and Growing Through Evaluation Module 5. https://www.cdc.gov/asthma/program_eval/asthmaprogramguide_mod5.pdf Cdc-pdf[PDF – 2 MB]

How to use TAARS

To use TAARS, follow the steps below, collecting the relevant information at the level of detail that fits your context and your needs. Tailor it to your program’s specifics.

As with all evaluation processes, begin by engaging the stakeholders who have an interest in the referral system. These may include service providers, case managers, clients, and those involved in recording and tracking referrals. In addition to improving your evaluation, stakeholder participation in the assessment will raise awareness about the importance of referrals, motivate reflection, and generate interest in strengthening the referral system.

1. Engage stakeholders

Engage stakeholders in determining the purpose for evaluating the referral system. The assessment can serve to:

• Identify what processes are used to document referrals
• Identify what works well and what needs improvement
• Establish a formalized referral network of service providers
• Assess written protocols and identify any need for modifications
• Strengthen the mechanisms for making and tracking referrals and examine efficiency
• Determine how well services are integrated
• Determine whether all patient populations are equitably served through the referral system

2. Describe the referral system

Describe the referral system by:

• Defining the coverage and scope of the referral system. (Does it cross health sectors? Does it cover sectors beyond health, such as education, housing, social services, etc.?)
• Making a list of the range of services offered; document services that are needed but not offered or available.
• Describing the expected referral pathways, describing which services should refer to which other services and documenting areas of expected bi-directionality.

3. Include a visual depiction

In addition to a narrative description, a flow chart or other graphic of how the referral system is structured may be helpful by detailing the universe of services and direction or bi-directionality of the referrals. Once you have a visual depiction or graphic, engage stakeholders to ensure there is shared understanding of the referral system and how it is intended to work.

Figure 1: The above diagram depicts the theoretical concept behind Rhode Island’s Comprehensive Integrated Asthma Control System and the intended linkages. This graphic illustrates the intricate bi-directional connections among programs and defines the range of the network linking asthma patients and their families to services. Providers within each of RI’s core asthma programs (HARP, BEAH, CASE, DAB) refer asthma patients to programs based on pre-determined eligibility and, in turn, receive information back from these programs, which encompass home, school, clinical, and community environments. Evaluation of the referral system tests the flow of information illustrated by the arrows.

4. Identify evaluation questions

Once you have a shared understanding of the referral system, engage in discussion to identify the evaluation questions that will effectively address the evaluation purpose. For example, you may want to know whether the protocols for the referral system are adequate and contain sufficient information, or if the referral system captures referrals in a timely manner such that it facilitates appropriate interventions. Be aware that reviewing the referral system protocols might elicit additional important evaluation questions you will want to answer.

Consider what your stakeholders need to know and how the information will be used. Tailor the design of your data collection efforts accordingly, bearing in mind the issue of “credibility” (i.e., evidence your stakeholders will value as valid and reliable.) Align efforts to your specific needs.

5. Gather and review referral system protocols and other supporting documentation

Determine if protocols or other guidance documents exist for referral system operations and expectations. Carefully review the protocols and all supporting documentation for pertinent information, noting any critical missing pieces. See Table 1 for a sample review template.

Table 1: Sample Template: Document review for referral system assessment

Table 1: Sample Template: Document review for referral system assessment

System Aspect	Documents	Issues to consider
Characteristics of referral network and system	Referral protocols Referral guidelines	Do referral protocols exist? (If not, what is the feasibility of developing a protocol? Of establishing a bi-directional referral system? What are the constraints?) Are protocols widely available to service providers within the network (i.e., at each facility or organization)? -Are protocols clinically sufficient? Do protocols cover the full range of relevant services? Do protocols reveal gaps in available resources and, if so, are measures being taken to fill those gaps? Do protocols include documenting how referrals are used (e.g., linkages to services; decision support; etc.) and type of settings they occur in (hospital, FQHC, pharmacy, etc.) Do protocols clearly describe how to track and document referrals? Do protocols capture the bi-directionality of referrals? Do protocols capture the method of referral delivery (e.g., in-person one-on-one, in-person group, electronic, phone, etc.)? Do protocols describe how to ensure client confidentiality? Are client preferences and logistic needs included in the referral protocol? Is everyone in the network using the same protocol?
	Directory of Network Services (listing of organizations providing related services in the geographic unit)	Do directories exist in relevant geographic areas or districts? Do directories include current contact information, location, hours of operation, and other relevant information? Are directories regularly updated? Are needed programs and services available in the relevant geographic unit (districts)?
	Formal agreement between referring and receiving institutions (for each organization and referring pathway)	Are providers in the network formally linked (i.e., formal agreements exist across services)? Do agreements cover the full range of relevant services? Do agreements specify processes for initiating referrals? Do agreements contain information regarding documentation of referrals? Do agreements specify total duration of referred program or service? (e.g., 6-month program, one time appointment, etc.)
Systems for tracking bi-directionality of referrals	Examples of registers, referral forms or tools used	Are the necessary data elements being collected by all facilities and organizations involved? Is bi-directionality of referrals captured and tracked? Is there consistency across providers on the type of information collected and the way it is collected? Is client confidentiality maintained?
	Report with compiled or analyzed referral data	Are data being collected and analyzed? Are recommendations for improvement proposed? Is referral-related information written and disseminated to stakeholders? Are reports and findings available at facilities or at the organizational level?
	Client satisfaction surveys	Are data on client satisfaction collected, analyzed and considered?
	Reports of data quality audits (DQA) of referral system data	Are data quality checks conducted routinely? Are data quality problems identified?
Other	Monitoring of referral system	Is regular monitoring of referral system occurring that informs adjustments to referral protocol, to referral system itself, communications, etc.?
	Report on evaluation of referral system	Has an evaluation of the referral system been conducted and when? What were the key findings of the evaluation, including secondary outcomes, such as client and staff perceptions of referral strategies, logistical issues, limitations, unanticipated benefits, potential harms, cost information, etc.?
	Evidence of training of service providers on referrals protocol	Did service providers receive training on the referral protocols? Are training materials available at the facility level?

6. Conduct interviews

Conduct interviews with key individuals, engaging stakeholders who have an interest in the services provided through the referral system.

• What are the services for which your organization refers clients elsewhere?
• How did you learn about the services and organizations you refer clients to?
• Please describe the method(s) and processes that are used to refer clients.
• Please describe the process for identifying client needs.
• Does your organization have a record keeping system to track referrals? If so, describe.
• What type of training on making referrals does staff receive?
• How does the provider at the receiving organization know that a patient has been referred to them?
• How does your organization know that a client has completed the referral?
• Is there a system to follow up with a client on a referral? If so please describe.
• Who usually follows up with a client on a referral?
• How is client confidentiality maintained throughout the referral process?
• Does your organization obtain permission form the client to follow up with other providers?
• What barriers have you encountered in making referrals for your clients?
• What suggestions do you have for improving the referral system or processes?

 

• Do you accept referrals from other services or organizations?
• What are the services for which clients are referred to your organization?
• Please describe how other providers know about the services that are provided by your organization.
• From where do you usually receive referrals?
• Please describe the mechanism or processes that are used to refer clients to your organization.
• Please describe the information you usually receive about a client who is referred to your organization.
• What information does your organization provide back to the provider or organization who made the referral?
• How has staff been trained in receiving and following up on referrals?
• How is client confidentiality maintained throughout the referral process?
• What barriers have you encountered in receiving client referrals?
• What suggestions do you have for improving the referral system or processes?
• What is your preferred method for receiving referrals

The following are additional questions for consideration regarding the referral system:

• Are referral data reported to anyone?
  • If so, how are these data collected and analyzed? Who is involved with reviewing and using this information?
• Are clients ever contacted regarding their experiences with the referral process?
  • If so, is information collected regarding their satisfaction and barriers to services? How are these data collected and analyzed? Who is involved with reviewing and using this information?

7. Analyze the data

Once data are collected, use a participatory process with data collectors and stakeholders to analyze and reflect on the data collected. Actively engage stakeholders in reviewing the findings, interpreting them, and contributing to discussion about possible explanations for the findings. Discuss the measures by which you will judge whether the referral system is “successful” and how you will justify your conclusions.
Consider the referral system elements in Table 4 below and tailor the list to your program’s characteristics and needs.

Table 2: Referral system elements to analyze.

Table 2: Referral system elements to analyze.

Elements	Strengths	Weaknesses
Referral protocols Available Appropriate to local context/needs Clear Up to date Training offered to staff
Outgoing referrals Occurring as in a timely manner Documentation/ registration proper, complete Information exchange between providers Processes to facilitate referral completion Training offered to staff
Incoming referrals Referral completion Documentation/ registration proper, complete Information exchange between providers Processes to facilitate follow-up Training offered to staff
Follow-up Documentation/registration proper Information exchange between providers Processes to follow-up with clients Training offered to staff
Referral documentation Data collection systems Data quality Confidentiality Training offered to staff
Data collection tools Compatibility of tools across service providers in the network Data elements are the same and similarly defined Training offered to staff
Use of referral data for decision making Referral data analyzed Recommendations made Facilities informed of results Evaluations conducted and disseminated Recommendations from network consortiums Training offered to staff

8. Ensure use of findings

Conclude with recommendations and action planning to ensure findings are used. After analyzing the referral system elements and interview responses, discuss with those engaged in the assessment and make recommendations for areas of improvement. Clearly document and share these recommendations with key referral system stakeholders (both at the local and state levels, as appropriate) and with key decision-makers who can effect changes in the referral system’s functioning. Prioritize recommendations and define concrete actions to improve the referral system. Develop an Action Plan to document plans for using the findings from the assessment. Refer to Appendix K of Implementing Evaluations (Module 2 of Learning and Growing through Evaluation) for a template specifically for documenting and tracking use of evaluation findings: https://www.cdc.gov/asthma/program_eval/asthmaprogramguide_mod2.pdf Cdc-pdf[PDF – 1 MB]

Acknowledgements

Maureen Wilce and Robin Kuwahara of CDC’s National Asthma Control Program Evaluation Technical Advisors would like to acknowledge the following individuals for graciously offering to review and provide feedback on TAARS: Anna Bradley, Evaluator, Health Improvement Section Chronic Disease Prevention & Health Promotion Bureau, Montana Department of Public Health & Human Services; Ann Connelly, Public Health Nurse Supervisor, Ohio Department of Health, Bureau of Maternal, Child and Family Health, School Nursing and Asthma Programs; Meridith Mueller, Program Evaluation Specialist, University of Wisconsin, Madison School of Medicine and Public Health, Department of Population Health Sciences, Wisconsin Division of Public Health; and Julian Rodriguez-Drix and Deborah Pearlman, Program Manager and Epidemiologist/Evaluator, respectively, for the Rhode Island Asthma Control Program.

References and Resources

Referral Systems Assessment and Monitoring Toolkit, MEASURE Evaluation, 2013
https://www.measureevaluation.org/resources/publications/ms-13-60External

Learning & Growing through Evaluation: State Asthma Program Evaluation Guide,
https://www.cdc.gov/asthma/program_eval/guide.htm

Community-Clinical Linkages for Prevention Guide for Implementation,
https://www.cdc.gov/dhdsp/pubs/docs/ccl-practitioners-guide.pdf Cdc-pdf[PDF, 39 pages]

Community-Clinical Linkages (Agency for Healthcare Research and Quality [AHRQ]):
http://www.ahrq.gov/professionals/prevention-chronic-care/improve/community/External

Building Relationships Between Clinical Practices and the Community to Improve Care (AHRQ):
https://innovations.ahrq.gov/topic-collections/building-relationships-between-clinical-practices-and-community-improve-careExternal

Delivering Preventive Services Through Clinical and Community Linkages: An Interview With Ruta Valaitis RN, PhD, Dorothy C. Hall Chair in Primary Health Care Nursing, Associate Professor, School of Nursing, McMaster University (AHRQ): https://innovations.ahrq.gov/perspectives/delivering-preventive-services-through-clinical-community-linkagesExternal

Integrating Evidence-Based Clinical and Community Strategies to Improve Health (U.S. Preventive Services Task Force): https://www.uspreventiveservicestaskforce.org/Page/Name/integrating-evidence-based-clinical-and-community-strategies-to-improve-healthExternal

Community-Clinical Relationships Measures Atlas (AHRQ):
https://www.ahrq.gov/sites/default/files/publications/files/ccrmatlas.pdf Cdc-pdf[PDF, 145 pages]External

Community-Clinical Relationships Evaluation Roadmap (AHRQ):
https://www.ahrq.gov/sites/default/files/publications/files/ccreroadmap.pdf Cdc-pdf[PDF, 43 pages]External