Tool for Assessing Asthma Referral Systems
- How to use TAARS
- 1. Engage stakeholders
- 2. Describe the referral system
- 3. Include a visual depiction
- 4. Identify evaluation questions
- 5. Gather and review referral system protocols and other supporting documentation
- Table 1: Sample Template: Document review for referral system assessment
- 6. Conduct interviews
- Sample questions for key stakeholders who make referrals
- Sample questions for key stakeholders who receive referrals
- 7. Analyze the data
- 8. Ensure use of findings
- References and Resources
The Tool for Assessing Asthma Referral Systems (TAARS) was developed by evaluators in the Centers for Disease Control and Prevention (CDC) National Asthma Control Program. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the CDC.
Recommended citation: Tool for Assessing Asthma Referral Systems (TAARS). Atlanta, GA: Centers for Disease Control and Prevention, National Center for Environmental Health, Division of Environmental Health and Health Effects, Air Pollution and Respiratory Health Branch, October 2017.
The Tool for Assessing Asthma Referral Systems (TAARS) is intended for use by asthma control programs as a guide in helping to understand the how effectively their referral systems are operating within their programs. Programs can use TAARS to conduct a comprehensive or more focused assessment of their referral system. Assessment may include determining how a referral system is structured, how networks are formed, whether appropriate written referral protocols and guidelines exist, the processes providers follow to refer clients, how well referrals are tracked and followed up, and barriers to referral initiation and referral completion. Using TAARS can also help you better understand how the different types of referrals among the different service providers operate in communities.
TAARS was adapted from the Referral Systems Assessment and Monitoring Toolkit, originally developed by MEASURE Evaluation for the President’s Emergency Plan for AIDS Relief (PEPFAR) for HIV/AIDS service providers. This tool can be used as is or it can be adapted to the specific design and purpose of your particular asthma referral system. It may be most useful in the context of overall evaluation planning and implementation, as recommended in the Learning and Growing through Evaluation series (https://www.cdc.gov/asthma/program_eval/guide.htm). Specifically, TAARS can help guide the focus and methodological considerations of a broader evaluation, such as is laid out in Steps 3 and 4 (Focus the Evaluation Design and Gather Credible Evidence) of the CDC’s Framework for Program Evaluation (https://www.cdc.gov/asthma/program_eval/guide.htm).1
Integrating health services across clinical and community-based settings improves the efficiency of the health system and provides greater access to services for clients, ultimately improving the well-being of clients. Providing comprehensive asthma control services entails integration and linkages between clinical services and services that exist outside of the health system, such as social services, educational services, and community-based services. For this reason, referral systems, which may also be called linkage systems, should be efficient, effectively connect people to needed services, and provide sufficient data to assure optimal care management.
Community services sites often use their own unique, sometimes entirely unstructured, referral networks. These networks may rely solely on informal, personal recommendations. In these settings, there is not just one referral system but several unstructured, very fluid systems operating simultaneously. A tool like TAARS can help you better understand the nuances of those systems and help identify where the gaps are and where improvements can be made. TAARS can also be useful in developing contracts or agreements with partners, describing the processes each uses for making or receiving referrals to or from programs, and to potentially encourage some level of standardization or consistency across programs. Further, and importantly, documentation of the context for the referral system will promote understanding and communication across programs and serve as institutional memory, especially valuable when staff turnover occurs.
1 For additional guidance on evaluating referrals related to asthma home visiting programs. Please see Appendix E of Evaluation of Services and Health Systems Interventions, Learning and Growing Through Evaluation Module 5. https://www.cdc.gov/asthma/program_eval/asthmaprogramguide_mod5.pdf Cdc-pdf[PDF – 2 MB]
To use TAARS, follow the steps below, collecting the relevant information at the level of detail that fits your context and your needs. Tailor it to your program’s specifics.
As with all evaluation processes, begin by engaging the stakeholders who have an interest in the referral system. These may include service providers, case managers, clients, and those involved in recording and tracking referrals. In addition to improving your evaluation, stakeholder participation in the assessment will raise awareness about the importance of referrals, motivate reflection, and generate interest in strengthening the referral system.
Engage stakeholders in determining the purpose for evaluating the referral system. The assessment can serve to:
- Identify what processes are used to document referrals
- Identify what works well and what needs improvement
- Establish a formalized referral network of service providers
- Assess written protocols and identify any need for modifications
- Strengthen the mechanisms for making and tracking referrals and examine efficiency
- Determine how well services are integrated
- Determine whether all patient populations are equitably served through the referral system
Describe the referral system by:
- Defining the coverage and scope of the referral system. (Does it cross health sectors? Does it cover sectors beyond health, such as education, housing, social services, etc.?)
- Making a list of the range of services offered; document services that are needed but not offered or available.
- Describing the expected referral pathways, describing which services should refer to which other services and documenting areas of expected bi-directionality.
In addition to a narrative description, a flow chart or other graphic of how the referral system is structured may be helpful by detailing the universe of services and direction or bi-directionality of the referrals. Once you have a visual depiction or graphic, engage stakeholders to ensure there is shared understanding of the referral system and how it is intended to work.
Figure 1: The above diagram depicts the theoretical concept behind Rhode Island’s Comprehensive Integrated Asthma Control System and the intended linkages. This graphic illustrates the intricate bi-directional connections among programs and defines the range of the network linking asthma patients and their families to services. Providers within each of RI’s core asthma programs (HARP, BEAH, CASE, DAB) refer asthma patients to programs based on pre-determined eligibility and, in turn, receive information back from these programs, which encompass home, school, clinical, and community environments. Evaluation of the referral system tests the flow of information illustrated by the arrows.
Once you have a shared understanding of the referral system, engage in discussion to identify the evaluation questions that will effectively address the evaluation purpose. For example, you may want to know whether the protocols for the referral system are adequate and contain sufficient information, or if the referral system captures referrals in a timely manner such that it facilitates appropriate interventions. Be aware that reviewing the referral system protocols might elicit additional important evaluation questions you will want to answer.
Consider what your stakeholders need to know and how the information will be used. Tailor the design of your data collection efforts accordingly, bearing in mind the issue of “credibility” (i.e., evidence your stakeholders will value as valid and reliable.) Align efforts to your specific needs.
Determine if protocols or other guidance documents exist for referral system operations and expectations. Carefully review the protocols and all supporting documentation for pertinent information, noting any critical missing pieces. See Table 1 for a sample review template.
|System Aspect||Documents||Issues to consider|
|Characteristics of referral network and system||Referral protocols
|Directory of Network Services (listing of organizations providing related services in the geographic unit)||
|Formal agreement between referring and receiving institutions (for each organization and referring pathway)||
|Systems for tracking bi-directionality of referrals||Examples of registers, referral forms or tools used||
|Report with compiled or analyzed referral data||
|Client satisfaction surveys||
|Reports of data quality audits (DQA) of referral system data||
|Other||Monitoring of referral system||
|Report on evaluation of referral system||
|Evidence of training of service providers on referrals protocol||
Conduct interviews with key individuals, engaging stakeholders who have an interest in the services provided through the referral system.
- What are the services for which your organization refers clients elsewhere?
- How did you learn about the services and organizations you refer clients to?
- Please describe the method(s) and processes that are used to refer clients.
- Please describe the process for identifying client needs.
- Does your organization have a record keeping system to track referrals? If so, describe.
- What type of training on making referrals does staff receive?
- How does the provider at the receiving organization know that a patient has been referred to them?
- How does your organization know that a client has completed the referral?
- Is there a system to follow up with a client on a referral? If so please describe.
- Who usually follows up with a client on a referral?
- How is client confidentiality maintained throughout the referral process?
- Does your organization obtain permission form the client to follow up with other providers?
- What barriers have you encountered in making referrals for your clients?
- What suggestions do you have for improving the referral system or processes?
- Do you accept referrals from other services or organizations?
- What are the services for which clients are referred to your organization?
- Please describe how other providers know about the services that are provided by your organization.
- From where do you usually receive referrals?
- Please describe the mechanism or processes that are used to refer clients to your organization.
- Please describe the information you usually receive about a client who is referred to your organization.
- What information does your organization provide back to the provider or organization who made the referral?
- How has staff been trained in receiving and following up on referrals?
- How is client confidentiality maintained throughout the referral process?
- What barriers have you encountered in receiving client referrals?
- What suggestions do you have for improving the referral system or processes?
- What is your preferred method for receiving referrals
The following are additional questions for consideration regarding the referral system:
- Are referral data reported to anyone?
- If so, how are these data collected and analyzed? Who is involved with reviewing and using this information?
- Are clients ever contacted regarding their experiences with the referral process?
- If so, is information collected regarding their satisfaction and barriers to services? How are these data collected and analyzed? Who is involved with reviewing and using this information?
Once data are collected, use a participatory process with data collectors and stakeholders to analyze and reflect on the data collected. Actively engage stakeholders in reviewing the findings, interpreting them, and contributing to discussion about possible explanations for the findings. Discuss the measures by which you will judge whether the referral system is “successful” and how you will justify your conclusions.
Consider the referral system elements in Table 4 below and tailor the list to your program’s characteristics and needs.
Table 2: Referral system elements to analyze.
|Data collection tools
|Use of referral data for decision making
Conclude with recommendations and action planning to ensure findings are used. After analyzing the referral system elements and interview responses, discuss with those engaged in the assessment and make recommendations for areas of improvement. Clearly document and share these recommendations with key referral system stakeholders (both at the local and state levels, as appropriate) and with key decision-makers who can effect changes in the referral system’s functioning. Prioritize recommendations and define concrete actions to improve the referral system. Develop an Action Plan to document plans for using the findings from the assessment. Refer to Appendix K of Implementing Evaluations (Module 2 of Learning and Growing through Evaluation) for a template specifically for documenting and tracking use of evaluation findings: https://www.cdc.gov/asthma/program_eval/asthmaprogramguide_mod2.pdf Cdc-pdf[PDF – 1 MB]
Maureen Wilce and Robin Kuwahara of CDC’s National Asthma Control Program Evaluation Technical Advisors would like to acknowledge the following individuals for graciously offering to review and provide feedback on TAARS: Anna Bradley, Evaluator, Health Improvement Section Chronic Disease Prevention & Health Promotion Bureau, Montana Department of Public Health & Human Services; Ann Connelly, Public Health Nurse Supervisor, Ohio Department of Health, Bureau of Maternal, Child and Family Health, School Nursing and Asthma Programs; Meridith Mueller, Program Evaluation Specialist, University of Wisconsin, Madison School of Medicine and Public Health, Department of Population Health Sciences, Wisconsin Division of Public Health; and Julian Rodriguez-Drix and Deborah Pearlman, Program Manager and Epidemiologist/Evaluator, respectively, for the Rhode Island Asthma Control Program.
Referral Systems Assessment and Monitoring Toolkit, MEASURE Evaluation, 2013
Learning & Growing through Evaluation: State Asthma Program Evaluation Guide,
Community-Clinical Linkages for Prevention Guide for Implementation,
https://www.cdc.gov/dhdsp/pubs/docs/ccl-practitioners-guide.pdf Cdc-pdf[PDF, 39 pages]
Community-Clinical Linkages (Agency for Healthcare Research and Quality [AHRQ]):
Building Relationships Between Clinical Practices and the Community to Improve Care (AHRQ):
Delivering Preventive Services Through Clinical and Community Linkages: An Interview With Ruta Valaitis RN, PhD, Dorothy C. Hall Chair in Primary Health Care Nursing, Associate Professor, School of Nursing, McMaster University (AHRQ): https://innovations.ahrq.gov/perspectives/delivering-preventive-services-through-clinical-community-linkagesExternal
Integrating Evidence-Based Clinical and Community Strategies to Improve Health (U.S. Preventive Services Task Force): https://www.uspreventiveservicestaskforce.org/Page/Name/integrating-evidence-based-clinical-and-community-strategies-to-improve-healthExternal
Community-Clinical Relationships Measures Atlas (AHRQ):
https://www.ahrq.gov/sites/default/files/publications/files/ccrmatlas.pdf Cdc-pdf[PDF, 145 pages]External
Community-Clinical Relationships Evaluation Roadmap (AHRQ):
https://www.ahrq.gov/sites/default/files/publications/files/ccreroadmap.pdf Cdc-pdf[PDF, 43 pages]External