Skip to main content

Documenting HIV research-utilization activities, outputs and outcomes: examples and lessons learned from Project SOAR

Abstract

The importance of using research findings to inform policy and program decisions is well recognized, but the literature on measuring research utilization activities is scarce. As funding to support some areas of research wanes or remains stagnant, the need to document the value of investing in research by its’ effect on improved programs and policies becomes increasingly necessary. We present the experience of Project SOAR, a six-year USAID-funded project focusing on HIV/AIDS-related implementation research, to demonstrate measurement of research utilization. We follow the project’s research-utilization logic model, including inputs, activities, outputs, and outcomes. We present tools the project developed and examples from project studies and discuss what works, remaining challenges and how to overcome them, and lessons learned. We then make recommendations for incorporating research-utilization activities and measurement in implementation-research studies.

Peer Review reports

Background

Research utilization is “the implementation of research-based knowledge (science) in practice” [1]. The United States Agency for International Development (USAID) supports implementation research to identify, develop and measure the impact of innovative strategies to improve public-health service delivery, and to inform policies and programs [2]. However, the link between research findings and public-health programs is often weak. While public-health services and programs usually claim to be evidence based, frequently relevant research findings are not considered in their design and implementation, and some studies are undertaken without considering programmatic implications [3]. Research utilization is the process of synthesizing research findings and using them to inform policy or program decision making.

The importance of research utilization is well recognized, and there is substantial literature documenting the implementation of research-utilization activities. However, not much is known about measuring the outcomes of such activities [4]. Validated tools for measuring research use are scarce. In a literature review of measuring research utilization, Estabrooks and Wallin [1] noted that researchers who have measured research use have tended to develop their own tools. Straus et al. [3] cite a comprehensive literature review and conclude that the most common methods for measuring research utilization rely on self-reporting by investigators or stakeholders.

This manuscript adds to the limited but growing body of knowledge on how to measure outcomes of research utilization, by presenting research-utilization activities undertaken by Project SOAR (Supporting Operational AIDS Research) a six-year implementation-science project (2014–2020), funded by USAID. The project was designed to improve HIV service delivery by conducting high quality research to meet data needs of stakeholders, strengthen the capacity of stakeholders to conduct implementation-science research and use study findings to guide planning, funding and program implementation. The project consisted of more than 50 research studies in 24 countries, focusing on implementation science to improve HIV prevention, care and treatment services. Research utilization was embedded in all of these studies and was incorporated in all phases of study implementation, starting with the study design. That is, studies were designed with the end in mind – they were designed to yield data that will be of use to policy decision makers.

We document how the project’s research-utilization activities were measured, and lessons learned in documenting inputs, outputs, and outcomes of research-utilization activities, including key challenges and recommendations for overcoming them. We do so by following the project logic model.

Measuring Project SOAR use of research findings

Project SOAR research-utilization logic model

Because the focus of Project SOAR was implementation science and the utilization of project research findings, the project developed a research-utilization logic model – a road map for those project activities that specifically focus on research utilization. In developing the research-utilization activity logic model, Project SOAR utilized The Centers for Disease Control and Prevention (CDC) guidelines for logic models of evaluation programs designed to address sexually transmitted infections (STI). The guidelines outlined the following components of a logic model [5]:

  1. a)

    Inputs (resources): funding, staff, materials;

  2. b)

    Activities (program, events or strategies): staff training, patient testing and treatment;

  3. c)

    Outputs (products of activities): number of patients treated, quality of training;

  4. d)

    Short-term outcomes (intended effects that occur within weeks or months): changes in knowledge, skills, or beliefs; increased proportion of patients treated;

  5. e)

    Intermediate outcomes (intended effects that occur over the mid-term: months-years): changes in policies or behaviors, increased proportion of sexual partners treated, increased condom use; and

  6. f)

    Long term outcomes or impact (long term intended effects: years or decades): reduced STI prevalence; changes in morbidity or mortality

While CDC’s logic model guidelines were intended for use in monitoring and evaluating STI programs (not research), these components are widely accepted as key elements of most logic models. Therefore, Project SOAR adopted the CDC’s logic model, but modified it to fit a different purpose. Project SOAR’s inputs into research-utilization activities are comprised of staff time, funding for travel, meetings and workshops; and guidance documents and tools to facilitate activity implementation. Outputs of these activities are the number of data use plans, meetings and dissemination events conducted by study teams to engage stakeholders in study implementation and dissemination of results. Outcomes are defined as the use of study findings, by stakeholders, to improve services, service guidelines and policies, with the ultimate long-term impact being improvements in the Joint United Nations Programme for HIV/AIDS (UNAIDS) 90–90-90 goals, which call for 90% of HIV-infected individuals to be diagnosed by 2020, 90% of whom will be on anti-retroviral therapy (ART) and 90% of whom will achieve sustained virologic suppression. The Project SOAR research-utilization logic model is shown in Fig. 1.

Fig. 1
figure1

Logic model for Project SOAR research utilization activities

Measuring research-utilization inputs

Research-utilization inputs include staff, budget, and resource materials. Project SOAR employed three staff persons dedicated to research utilization:

  1. 1.

    A full time Research-Utilization Advisor, who served as a dedicated knowledge broker;

  2. 2.

    A full time Science Writer; and

  3. 3.

    A Knowledge Management Specialist, who worked on research utilization 50% of time.

Their roles are described in the section on Activities below. With guidance from these dedicated staff, all Project SOAR’s principal investigators and staff considered research utilization in all aspects of their research studies. For each study, the principal investigator included a budget allotment for research utilization activities. In addition, at the global level, the budget included allotments for capacity strengthening workshops, small grants for research utilization, as well as travel for the Research-utilization advisor, Science Writer and Knowledge Management Specialist.

The project developed and used a Research-Utilization Process Guide, which provided guidelines and tools to use by project partners, researchers, and other staff [6].

Measuring research-utilization activities

By definition, research utilization involves policy makers and other stakeholders, such as government officials, program managers, academia and community leaders, who may use research findings to inform policies and programs, and who may have need for such data. While engaging key stakeholders in study design, and sharing knowledge of study findings with them, may not be effective on their own to change practices or policies, they are necessary pre-requisites for change [4]. Therefore, Project SOAR conducted activities to strengthen the capacity of stakeholders to generate, analyze and use research findings. During site-selection exploratory visits to study countries, each study’s principal investigator identified in-country colleagues to act as co-principal investigators on the studies, as well as key stakeholders to help refine research questions to address priority policy and program needs in the country, and undertake capacity-building activities, to promote their demand for and use of research data. One such activity was convening two regional capacity-strengthening workshops in Johannesburg, South Africa in February, 2017 and May, 2018. The workshops were attended by 48 key stakeholders from 12 countries in which the project was active, and focused on skill building around demand for research data and use of findings.

To enable workshop participants to apply the skills they learned to generate and use data, Project SOAR posted a request for proposals to in-country researchers and issued small grants of not more than $10,000 each to nine of the 16 applications received. Using these funds, in-country researchers conducted secondary analyses of available data to address locally relevant research questions, convened stakeholder meetings to disseminate findings, and submitted conference abstracts and journal manuscripts.

While studies funded through the small grants’ mechanism yielded interesting and utilized findings, the main focus of Project SOAR was the studies undertaken directly by the project. The project recognized the importance of engaging stakeholders in all stages of each of these studies, from design, through implementation, to ensure that the research question has policy and program relevance, and to maximize the eventuality that findings are used to inform activities. Therefore, the engagement with stakeholders started in the inception phase of each study, during which the study design and methodology were discussed, and continued into later phases of the study. During study implementation researchers shared preliminary, and eventually final, study findings with stakeholders, and brainstormed with them implications of the findings to current policies and programs.

To document stakeholder engagement in each study, Project SOAR developed a Stakeholder Engagement Template. An example of how the template was used is shown in Table 1, in which the template is populated with information from the Lesotho IMPROVE study [7]. The template was completed by the principal investigator of each Project SOAR study and submitted to project management twice per year, as part of the project management and monitoring. The template describes all stakeholder engagement efforts for the study, including which stakeholders were engaged, dates and modes of engagement, as well as what was discussed during the engagement.

Table 1 Stakeholder engagement template, Lesotho IMPROVE study [7]

Researchers also engaged stakeholders through existing technical working groups or study-specific research advisory committees, using presentations and results briefs to present study findings. Each Project SOAR study conducted one research advisory committee meeting at inception, another to share preliminary study results, followed by a data interpretation meeting to discuss final study findings and determine study recommendations, before conducting a final study dissemination meeting, attended by a broader group of stakeholders. In each case the researchers reported to Project SOAR management the key aspects of the meeting using the template in Table 1. The data in the last column of the table, or outcomes of each stakeholder engagement, then flowed directly into outcome monitoring, described later. Further details of the process of engagement of stakeholders in Project SOAR studies are provided in a previous publication [8].

The Research-Utilization Advisor also provided technical support to study teams by way of country visits, phone calls and e-mails, and supported the formation of country-level research advisory committees in each of the countries where Project SOAR was active. The technical support included review of study proposals, meeting plans, agendas, and power point presentations; as well as assisting in selecting the types of stakeholders to be engaged. By the end of the fifth project year, Project SOAR had established 54 research advisory committees and supported them in developing 46 data-use plans in 24 countries. Members of Project SOAR’s scientific team supported in-country teams by reviewing their draft publications and by mentoring small grant recipients. The Research-utilization advisor, the Science Writer and the Knowledge Management Specialist also supported the researchers in editing, formatting and preparing publications, ensuring that the publications highlighted programmatic and policy implications of study findings.

Other activities undertaken by the Research-Utilization Advisor included promoting research utilization among project researchers and staff and providing technical assistance to monitoring how project findings had been utilized for program and policy change. He provided technical input to study protocols to ensure that study outcomes were aligned with salient issues in programs and policies of the country where the research was being conducted, and collaborated with principal investigators in study protocol-development and trips to study countries, where they scoped the landscape of stakeholders and policy issues relevant to the research topic of interest.

The Research-Utilization Advisor worked with researchers to strengthen the capacity of in-country investigators and governmental stakeholders to access, review and use research to improve programs and policies through workshops and small grants. In addition, the advisor and Project SOAR’s Science Writer and Knowledge Management Specialist worked with researchers to analyze the data, identify key findings, develop practical recommendations and create power point presentations, activity briefs and results briefs that researchers used to disseminate their research and results. Table 2 summarizes research-utilization activities of Project SOAR through the end of the fifth project year.

Table 2 Project SOAR activities to facilitate research utilization

Measuring research-utilization outputs

As shown above, Project SOAR conducted technical support and capacity strengthening activities directed at in-country research teams and stakeholders, to enable them to carry out research-utilization activities. Outputs of these activities were progressively monitored for each study as it was ongoing, then tabulated project wide. The outputs for Project SOAR through the end of the fifth project year are shown in Table 3. These outputs were directly linked to Project SOAR’s inputs and activities.

Table 3 Outputs: Project SOAR research utilization outputs

The research advisory committees developed data-use plans that they used to engage stakeholders throughout each study as living documents, modified as they gained more knowledge about the landscape of stakeholders and priority program and policy issues. Final data-use plans, developed at the final dissemination meeting of each study, were plans for continued engagement of stakeholders after the close of Project SOAR. As part of the ongoing data-use plan, research advisory committees identified and coached program and policy influencers, referred to as champions, to continue engaging stakeholders in various forums to integrate study findings into decision making processes.

As shown in Table 3, Project SOAR researchers developed 58 Activity Briefs and 74 Results Briefs. Briefs are two-page documents used to disseminate study information to stakeholders. Activity briefs focus on one study, stating the study objectives, methods, and proposed research-utilization process; Result briefs focus on key findings, programmatic implications and recommendations. The briefs were printed and shared with stakeholders as hard copies and were also published on the Project SOAR website.

In the six countries where the project had multiple studies, Project SOAR researchers conducted 11 joint research advisory group meetings that brought together an average of 40 study staff and key stakeholders in each country, to interpret preliminary and final study findings, identify key messages, and make practical and actionable recommendations that applied study findings to strengthen policies and programs. In countries where it was not possible for national authorities to attend dissemination meetings, Project SOAR researchers requested boardroom meetings at national AIDS program offices, where they shared study findings with top-level government officials and discussed policy and programmatic implications of the findings.

Beyond national meetings, Project SOAR researchers shared interim research findings in 116 webinars and meetings convened by USAID and other stakeholders, and in 111 oral and poster presentations in regional and international AIDS conferences. They also submitted 59 manuscripts for publication in peer reviewed journals, to further disseminate study findings. As Table 3 shows, the project compiled and shared best practices of the project’s research-utilization approach to communities of practice through meetings and publications [8,9,10,11].

Measuring research-utilization outcomes

Outcomes of stakeholder engagement are incidents where study findings are used to improve policies and programs. The literature suggests at least three domains of research use, where ‘use’ refers to an event or action by stakeholders to change programs or policies based on study findings [4, 12, 13]. Instrumental Use is when stakeholders use research to make policy or program decisions. Conceptual Use is when stakeholders apply the new knowledge in their thinking and conceptual understanding of the issue [13] and use study findings in debates and ‘public and professional discourses’ [12] without necessarily taking action to change policy or practice as a result of the study findings. Finally, Persuasive Use is when stakeholders use research data to influence or persuade other stakeholders, such as politicians to pass a bill or community members to change a behavior.

When Project SOAR researchers shared research information, updates on the research process, or study findings, some stakeholders made comments and asked questions to seek clarification, interpret the data and draw programmatic or policy implications. These productive interactions, or instances of knowledge exchange, are often viewed as crucial Conceptual Use that leads to Instrumental Use [14] and should be documented [4]. Indeed, Penfield et al. [15] recommend documentation of the bi-directional flow of knowledge between researchers and stakeholders. Project SOAR considered conceptual and persuasive use of research findings as short-term outcomes (weeks/months), and instrumental use as intermediate outcomes (months/years).

Project SOAR short-term outcomes

In the bi-directional exchange between researchers and stakeholders some stakeholders may commit to implement study recommendations. These commitments can be classified as Conceptual Use, even if they are only verbal, and it was important for Project SOAR study teams to record them as prompts for future follow up. Indeed, Project SOAR champions, identified in the data-use plans (Table 3) planned to follow up to verify whether the stakeholders fulfilled these commitments; and if there were barriers to fulfilling the commitments, how they could be addressed.

Several stakeholders stated a commitment to use the study intervention in some way, such as integration into routine service delivery, pilot-testing an intervention, or taking an intervention to scale based on research findings. Other stakeholders committed to addressing service gaps identified by the study, for example through improved program monitoring and evaluation or through training. These are summarized in Table 4 [16,17,18,19,20,21,22]. It is important to note that in some situations, while no policy or program change were affected at the time of reporting, Project SOAR recorded these commitments and considered them to be a step in the right direction. For example, in the Zambia project YES study, the integration of anti-stigma activities in the study intervention was a step towards developing an anti-stigma program [17]. Similarly, in the Kenya and Uganda Pediatric Case Finding studies, the development of quality improvement plans was a step towards improved programing [16].

Table 4 Project SOAR Short-term outcomes (weeks- months)

Other than commitments, another Conceptual Use of research can be an improvement in knowledge about a topic that feeds into policy debate. For example, the Uganda PEPFAR (President’s Emergency Plan For AIDS Relief) Geographical Pivot study showed that when PEPFAR transitioned out of some health facilities, the government was not immediately able to provide continuation of ART [18]. This finding elicited a debate among senior policy makers about the government’s preparedness for donor withdrawal.

Table 4 also shows examples of Persuasive Use [23, 24]. In the case of the Malawi DREAMS (Determined, Resilient, Empowered, AIDS-free, Mentored, and Safe) study, the Ministry of Health Department of HIV/AIDS epidemiologist intended to use study findings to influence the ongoing go/no-go discussions about Pre-Exposure Prophylaxis (PrEP) use in this population; and the organizations retrieving data from the Global Fund repository on key populations, intended to use the data to influence global policy and interventions [24].

Project SOAR intermediate outcomes

Some stakeholders made policy or program decisions and acted on these decisions based on Project Soar study findings. This is Instrumental Use. Table 4 shows five instances of instrumental use of Project SOAR study findings. For example, in Tanzania after learning that community-based provision of ART improved initiation and retention in treatment among key populations, the Ministry of Health authorized community-based ART services for these populations. All service delivery providers were informed of this change in a formal government circular. Similarly, in Senegal, the AIDS program added HIV self-testing to the national strategy based on findings from a Project SOAR feasibility study (Table 5) [25,26,27,28,29].

Table 5 Project SOAR Short-term outcomes (weeks- months

Project SOAR impact

The aim of Project SOAR studies was to strengthen programs that contribute to the global goals of ending the HIV epidemic through attainment of the 90–90-90 targets. The first 90 stands for ‘90% of all people living with HIV will know their HIV status’; the second stands for ‘90% of all people diagnosed with HIV infection will receive sustained antiretroviral therapy; and the third ‘90% of all people receiving antiretroviral therapy will have viral suppression’ [30]. Therefore, the long-term impact of Project SOAR studies would be changes in these proportions as a result of study findings: increases in the proportion of people who know their HIV status (first 90), increases in the proportion of people living with HIV who are on ART (second 90), and increases in the proportion of those on ART who are virally suppressed (third 90).

By definition, impact can only be observed after years or even decades of program work. There is a time lag between publication of research results and the utilization of these findings for policy or program change, and the corresponding reflection of these changes on population-based indicators. This makes it difficult to document the long-term impact of research utilization in the course of a six-year project [15, 31]. Therefore, a key limitation of this paper is that we are unable to present data on impact. In addition, population-level indicators of prevalence of knowledge of HIV status, ARV treatment, and viral-load suppression are impacted by other ongoing activities and programs. It is therefore challenging to attribute the influence of isolated research-utilization activities on overall indicator improvement. PEPFAR conducts population-based HIV impact assessments, to gauge and guide the entire HIV response in priority countries [32]. Therefore, assessing the long-term impact of Project SOAR research-utilization activities is beyond the scope of the project.

Quality and cost considerations

We enumerate the number of incidents of use of research findings; however, we do not discuss qualitative parameters, such as contextual factors influencing successful adoption of research, such as targeting the right stakeholders at the right time in the program cycle, and prevailing favorable political circumstances. While such a detailed discussion of each incident of use would be ideal, it is beyond the scope of this paper. However, further information about each reported incident of use is published on Project SOAR’s website and in peer reviewed journals, and we provide relevant citation throughout the text. We invite research organizations interested in replicating Project SOAR’s research-utilization interventions to review the Project SOAR publications we cite and learn about the program-specific contexts, bearing in mind that while these contexts may not be replicable, the concepts maybe transferable.

We also do not provide a costing of the research-utilization interventions that may have led to the research-use incidents that we report. Such information is not readily available because, as explained above, research-utilization activities were budgeted as integral components of each Project SOAR study, and it would require a detailed costing study to extract the research utilization costs. Such a study is beyond the scope of this paper. Nonetheless, the paper presents three major categories of illustrative budget items that could inform replication. One category is personnel, and here we mention a 100% Research-Utilization Advisor, a 100% Science Writer and a 50% Knowledge Management Specialist. The second category is inputs, listed in Table 2, including capacity strengthening workshops, a small grants program, international trips for the Research-Utilization Advisor, and research advisory committee meetings. A third budget category comprises briefs, presentations and meetings used to disseminate study outcome information, listed in Table 3. We believe these budgetary items can assist research programs to incorporate research-utilization activities into their research budgets.

Discussion and conclusion

Project SOAR is dedicated to implementation science. As such, the project’s science (research studies) was undertaken specifically to inform the implementation of programs and policies. Therefore, research utilization was embedded in all phases of Project SOAR studies, from design through implementation and dissemination of findings. This essay presented the process Project SOAR used for research utilization and how it was documented and measured. Our discussion followed the project’s research-utilization logic model, which includes inputs, activities, outputs, outcomes (short-term and intermediate) and impact – not of the research studies, but of the utilization of findings from these studies.

To capture activities, outputs, and outcomes, the project developed prospective tools, such as the stakeholder engagement tool (Table 1). Using prospective tools, completed as the research-utilization activities progressed, enabled the researchers to capture ‘indicators as they emerge’ [15]. Prospectively collected indicators are more powerful than retrospective indicators, collected after a study is completed, because retrospective indicators may be prone to recall bias [33]. Collecting indicator data prospectively also allowed researchers to interact with stakeholders earlier and have more opportunities for utilizing research findings for program and policy improvements.

Challenges to utilization of study findings

Research utilization, like any program activity, has cases of failure, or less than optimal success. In the case of research utilization this translates to instances where research findings are not used for decision making. Documentation of research use should also include non-use of findings and reasons why data were not used. Reasons for non-use vary and may include, for example, insufficiently sound research methodology, unconvincing study findings, infidelity of a tested intervention, failure of a tested pilot intervention, or instances where stakeholders are not optimally engaged in the research process or the interpretation of results [34].

From the stakeholder perspective, reasons for non-use of research findings can be that the findings do not align with a decision-making moment, that there is no mechanism within which to enact change, or that there is no budget to incorporate the change. This is one of the reasons to document research utilization in all phases of research and dissemination, to enable researchers and stakeholders to link research impact or non-impact to how data utilization was promoted [15].

Project SOAR required all study researchers to report on activities and stakeholder engagement every 6 months through the life of the project, thus providing information needed to further analyze what was more, or less, effective in generating research-findings use. Following is a discussion of some challenges that Project SOAR, and other research projects, could face in monitoring of research-utilization activities and outcomes.

Reporting bias

Documentation and monitoring of research utilization is susceptible to reporting bias. Most methods of data collection to monitor research utilization involve interviews with, or by, research generators and research users [14, 28, 35], and are therefore subject to recall bias as the respondents tend to recall only what worked. They may also be susceptible to social desirability bias, because respondents would want to please the interviewers, or they have a vested interest in showing the value of the investment made in research. In addition, the selection of instances of research utilization that are reported tends to be biased towards “high-impact rather than low-impact” research, based on the level of impact on programs or policies [14]. Routine reporting every 6 month, and supervision of the Research-Utilization Advisors in all project studies, helped reduce these biases in Project SOAR studies.

Attribution

Attributing a policy or program change to a specific research finding is difficult and often impossible. Research-utilization activities are usually not implemented as randomized controlled interventions; results of a study are simply presented to decision makers without an experimental design to exclude other factors that could influence decision-making. Such factors include political forces and other external policy influences, good fortune, other studies, and the magnitude of the problem being addressed by the research findings [14, 15, 26]. Further, there is usually no counterfactual; meaning that there is no way to tell what would have happened had the study not been conducted [29], or had the results not been presented to these particular stakeholders, at that specific time, in exactly that manner and setting [28].

The closest we can get to attribution is when the change in policy or practice is made after the dissemination of findings to the specific stakeholders who made the decision. For example, after Project SOAR investigators presented study results to the national AIDS program in Senegal that showed the feasibility of HIV self-testing [26], the national AIDS program included self-testing in the national HIV test-and-start strategy. We can posit that without the Project SOAR study, the national AIDS program may have still included self-testing in its strategy; on the other hand, it could also be argued that Project SOAR’s feasibility study provided confidence to the national AIDS program to implement self-testing.

Searles et al. (2016) recommend that as part of the study protocol-development process, researchers should assess the policy landscape related to the guidelines or policies that the study is intending to influence. Such information can ensure that the correct stakeholders are identified, and give support to the case for attribution when corroborated with the fact that the policy or program change was made after the research findings were disseminated to the specific stakeholders who made the change [4].

Unpredictable time lag

The time between completion of research activities and utilization of its findings varies widely. Some study results are used immediately after a study is completed, even before the results are published; or it may take months or even years for results to affect policy or program change. For example, in Project SOAR’s study on the feasibility of HIV test-and-start in Namibia [20], the implementing partner used the data before the results were disseminated at a national meeting. The implementing partner then reported that preliminary study data were very compelling, so they went ahead and implemented some of the recommendations (such as differentiated ART for youth and reduction in frequency of clinician consultations for stable patients).

However, quite often research findings are not used for years or even decades after they are published [28]. With the possibility of infinite time lag between publication and utilization of research findings [15] it becomes difficult to document when study results have been used because often, after the study ends, the interested parties do not have the resources to continue monitoring and documenting instances of utilization. It is, therefore, impossible to conclude that a study’s results have not, or will never be, utilized. For this reason, Project SOAR investigators developed data use plans (in collaboration with stakeholders) which include the identification and cultivation of champions who will continue promoting the use of study findings at opportune moments, even after the end of the project. For example, in Malawi, Project SOAR study of the DREAMS program found a high prevalence of genital herpes among adolescent girls and young women, which could influence the go/no-go decision about providing PrEP to beneficiaries. Our Champion planned to follow up by attending the prevention technical working group and discussing our results, together with other data used for making the go/no-go decision.

Recommendations

Funders of implementation research are increasingly seeking to understand the impact of research to assess returns on their investments [4]. Moreover, as funding to support some areas of research wanes or remains stagnant, the need to document the value of investing in research by its’ effect on improved programs and policies becomes increasingly important.

While monitoring of research utilization is not a widely established component of research, Project SOAR’s experience has shown that researchers can use a simple set of tools and guidance to record activities they conduct, to systematically ensure that stakeholders are aware of research findings and the intent and commitment from stakeholders to take programmatic or policy decisions. Based on Project SOAR’s experience, we propose the following recommendations:

  1. 1.

    Research organizations should require researchers to routinely use research-utilization guidance and reporting tools for all significant research investments;

  2. 2.

    At the design phase of each study, assess the relevant policy landscape, to ensure that the appropriate stakeholders are identified and engaged, and to build the case for attribution;

  3. 3.

    Collect and use stakeholder engagement and research-utilization data progressively, throughout all phases of the study, starting at the design phase. This can help improve stakeholder buy-in to research findings, development of appropriate recommendations and minimize bias; and

  4. 4.

    Implement systematic research-utilization activities, including identifying and coaching in-country champions to promote study findings over time and to develop data use plans that define a roadmap for research-utilization activities to guide the champions to promote study findings at future opportunities after study results are disseminated.

Availability of data and materials

Not applicable.

Abbreviations

ART:

Anti-retroviral therapy

CDC:

Centers for Disease Control and Prevention

DREAMS:

Determined, Resilient, Empowered, AIDS-free, Mentored, and Safe

PEPFAR:

President’s Emergency Plan For AIDS Relief

PrEP:

Pre-exposure prophylaxis

SOAR:

Supporting Operational AIDS Research

STI:

Sexually transmitted infections

UNAIDS:

Joint United Nations Programme for HIV/AIDS

USAID:

United States Agency for International Development

References

  1. 1.

    Estabrooks CA, Wallin L. Where do we stand on the measurement of research utilization? Paper prepared for the 4th Annual Knowledge Utilization Colloquia (KU04), Belfast, Northern Ireland. 2004. https://cloudfront.ualberta.ca/-/media/nursing/knowledge-utilization-studies-program/knowledge-utilization-colloquia/ku04/wallinbelfast.pdf

    Google Scholar 

  2. 2.

    United States Agency for International Development (USAID). Improving HIV and AIDS programming through implementation science. Washington, DC: USAID; 2014. https://www.usaid.gov/global-health/health-areas/hiv-and-aids/technical-areas/implementation-science

  3. 3.

    Mallonee S, Fowler C, Istre GF. Bridging the gap between research and practice: a continuing challenge. Brit Med J. 2006;12(6):357–9. https://doi.org/10.1136/ip.2006.014159.

    CAS  Article  Google Scholar 

  4. 4.

    Searles A, Chris DC, Attia J, et al. An approach to measuring and encouraging research translation and research impact. Health Res Policy Syst. 2016;14:60.

    Article  Google Scholar 

  5. 5.

    Centers for Disease Control (CDC). Practical use of program evaluation among sexually transmitted disease (STD) programs. Atlanta: Division of STD Prevention, National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention, Centers for Disease Control and Prevention; 2014. https://www.cdc.gov/std/program/pupestd.htm

  6. 6.

    Project SOAR. RU Guide. Project SOAR’s approach to research utilization, Project SOAR 2016. Washington, D.C.: Population Council; 2016. http://www.projsoar.org/wp-content/uploads/2017/12/ResUtilization_Guidance.pdf

    Google Scholar 

  7. 7.

    Project SOAR. Lesotho IMPROVE. Evaluating a multidisciplinary integrated management team intervention to improve maternal and child outcomes and HIV service uptake and retention in Lesotho. Project SOAR activity brief. Washington, DC: Population Council; 2018. http://www.projsoar.org/wp-content/uploads/2018/04/Lesotho_IMPROVE_Brief.pdf

    Google Scholar 

  8. 8.

    Kalibala S, Nutley T. Engaging stakeholders, from inception and throughout the study, is good research practice to promote use of findings. AIDS Behav. 2019. https://doi.org/10.1007/s10461-019-02574-w Published online 3 July 2019.

  9. 9.

    Kalibala S. Improving HIV programs: Developing in-country capacity to generate and use Data.” , 2018 https://www.usaid.gov/global-health/health-areas/hiv-and-aids/information-center/research-corner/project-soar

    Google Scholar 

  10. 10.

    Project SOAR. Q&A with Sam Kalibala, Director of Research Utilization, answers questions about SOAR’s strategy to promote research uptake. 2019. http://www.projsoar.org/news/kalibala-soar-ru-strategy/

    Google Scholar 

  11. 11.

    Kalbarczyk A, Davis W, Kalibala S, et al. Research capacity strengthening in Sub‑Saharan Africa: Recognizing the importance of local partnerships in designing and disseminating HIV implementation science to reach the 90–90–90 goals. AIDS Behav. 2019. Published online 16 May 2019. https://doi.org/10.1007/s10461-019-02538-0.

  12. 12.

    Sumner A, Crichton J, Theobald S, Zulu E, Parkhurst J. What shapes research impact on policy? Understanding research uptake in sexual and reproductive health policy processes in resource poor contexts. Health Res Policy Sy. 2011;9(Suppl 1):S3.

    Article  Google Scholar 

  13. 13.

    Raftery J, Hanney S, Greenhalgh T, Glover M, Blatch-Jones A. Models and applications for measuring the impact of health research: Update of a systematic review for the Health Technology Assessment Programme. Health Technol. 2016;20(76):1–254.

    Google Scholar 

  14. 14.

    Makkar SR, Brennan S, Turner T, Williamson A, Redman S, Green S. The development of SAGE: A tool to evaluate how policymakers engage with and use research in health policymaking. Res Evaluat. 2016;25(3):315–28.

    Article  Google Scholar 

  15. 15.

    Penfield T, Baker MJ, Scoble R, Wykes MC. Assessment, evaluations, and definitions of research impact: A review. Res Evaluat. 2014;23:21–32.

    Article  Google Scholar 

  16. 16.

    Project SOAR. Kenya and Uganda pediatric case finding. Active pediatric HIV case finding in Kenya and Uganda. Washington, D.C.: Population Council; 2018. http://www.projsoar.org/our-activities/keny-uganda-ped-cases/

    Google Scholar 

  17. 17.

    Project SOAR. Zambia Project YES. Youth living with HIV in Zambia: interpersonal violence, self-stigma, and viral suppression. Project SOAR Results Brief. Washington, DC: Population Council; 2019. http://www.projsoar.org/wp-content/uploads/2019/05/Zambia_YES_Interim_ResBrief.pdf

    Google Scholar 

  18. 18.

    Rodríguez DC, Paina L, Wilhelm J, et al. Evaluating the impact of PEPFAR’s geographic prioritization on centrally supported health facilities. Project SOAR final report. Washington, D.C.: Population Council; 2019. http://www.projsoar.org/wp-content/uploads/2019/03/KenyUg_PEPFARGeo_Report.pdf

    Google Scholar 

  19. 19.

    Settergren SK, Mujaya S, Rida W, et al. Cluster randomized trial of comprehensive gender-based violence programming delivered through the HIV/AIDS program platform in Mbeya Region, Tanzania: Tathmini GBV study. PLoS ONE. 2018;13(12):e0206074. https://doi.org/10.1371/journal.pone.0206074.

    Article  PubMed  PubMed Central  Google Scholar 

  20. 20.

    Project SOAR. Namibia TnS. Viral load testing: Room for improvement in Namibia’s antiretroviral treatment services. Project SOAR Results Brief. Washington, DC: Population Council; 2019. http://www.projsoar.org/wp-content/uploads/2019/01/Namibia_TnS_VL_ResBrief.pdf

    Google Scholar 

  21. 21.

    Project SOAR. Tanzania FSW/FP. Assessing family planning and safer conception needs and services among female sex workers living with HIV in Dar es Salaam. Project SOAR Final Report. Washington, DC: Population Council; 2018. http://www.projsoar.org/wp-content/uploads/2018/12/Tz_FSW_FP_FinalReport.pdf

    Google Scholar 

  22. 22.

    Lisa B, Musoke P, Etima M, et al. Increasing pediatric HIV disclosure to children in Uganda. Project SOAR Results Brief. Washington, DC: Population Council; 2017. http://www.projsoar.org/wp-content/uploads/2017/10/Uganda_DISCOKids_Brief.pdf

    Google Scholar 

  23. 23.

    Project SOAR. Global Fund TA. Strengthening capacity to use data to inform HIV responses for key populations. Project SOAR Activity Brief. Washington, DC: Population Council; 2018. http://www.projsoar.org/wp-content/uploads/2018/06/20180621_GlobalFundTA_ActBrief_Final.pdf

    Google Scholar 

  24. 24.

    Mensch BS, Erica SH. Rates of HIV and HSV-2 among young people in Machinga, Malawi. Project SOAR Results Brief. Washington, DC: Population Council; 2017. http://www.projsoar.org/wp-content/uploads/2017/12/Malawi_DREAMS_MSAS_Brief1.pdf

    Book  Google Scholar 

  25. 25.

    Project SOAR. Tanzania FSW-ART. Community-based delivery of antiretroviral treatment for female sex workers in Tanzania: high levels of initiation, use, and adherence. Project SOAR Results Brief. Washington, DC: Population Council; 2019. http://www.projsoar.org/wp-content/uploads/2019/02/Tz_FSW_Bsln_MidlineBrief.pdf

    Google Scholar 

  26. 26.

    Project SOAR. Senegal TnS. Can HIV Self-testing Help Reach Those at Risk for HIV and Not Accessing Traditional Testing Services in Senegal? Project SOAR Results Brief. Washington, DC: Population Council; 2019. http://www.projsoar.org/wp-content/uploads/2019/01/Senegal_TnS_Bsln_ResBrief.pdf

    Google Scholar 

  27. 27.

    Chouraya C, Ashburn K, Khumalo P, et al. Association of antiretroviral drug regimen with viral suppression in HIV-positive children on antiretroviral therapy in Eswatini. Pediatr Infect Dis J. 2019;38(8):835–9. https://doi.org/10.1097/INF.0000000000002347.

    Article  PubMed  Google Scholar 

  28. 28.

    Project SOAR. Eswatini PrEP modelling. Oral Pre-Exposure Prophylaxis Modeling Results: Eswatini, Results to inform Ministry of Health’s PrEP scale up. Project SOAR Results Brief. Washington, DC: Population Council; 2019. http://www.projsoar.org/wp-content/uploads/2019/02/Eswatini_PrEPModeling_ResBrief.pdf

    Google Scholar 

  29. 29.

    Project SOAR. Uganda PrEP modelling. Oral Pre-Exposure Prophylaxis Modeling Results: Uganda, Results to inform Ministry of Health’s PrEP scale up. Project SOAR Results Brief. Washington, DC: Population Council; 2019. http://www.projsoar.org/wp-content/uploads/2019/02/Uganda_PrEPModeling_ResBrief.pdf

    Google Scholar 

  30. 30.

    UNAIDS (2020). 90-90-90: An ambitious treatment target to help end the AIDS epidemic. https://www.unaids.org/en/resources/909090

    Google Scholar 

  31. 31.

    Frank C, Nason E. Health research: measuring the social, health and economic benefits. Can Med Assoc J. 2009;180(5):528–34.

    Article  Google Scholar 

  32. 32.

    Centers for Disease Control (CDC). New PHIA survey data show critical progress towards global HIV targets; 2019. https://www.cdc.gov/globalhivtb/who-we-are/events/world-aids-day/phia-surveys.html

    Google Scholar 

  33. 33.

    Straus SE, Tetroe J, Graham ID, Zwarenstein M, Bhattacharyya O, Shepperd S. Monitoring use of knowledge and evaluating outcomes. CMAJ. 2010;182(2):E94–8.

    Article  Google Scholar 

  34. 34.

    Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Sy. 2011;9:26 http://www.health-policy-systems.com/content/9/1/26.

    Article  Google Scholar 

  35. 35.

    Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38:65–76.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge Project SOAR Senior Management Team: Dr. Scott Geibel, Project Director; Dr. Eileen Yam, Deputy Project Director; Ms. Deborah Weiss, Director of Operations; Dr. Deanna Kerrigan, formerly of Joint Hopkins University (JHU), who guided the research-utilization activities and ensured support and participation of the entire research team. We would also like to acknowledge Ms. Ellen Weiss, the Science Writer and Ms. Sherry Hutchinson, the Knowledge Management Specialist both of whom worked tirelessly to analyze, edit, format and produce research-utilization resources and results dissemination materials and products. In addition, we acknowledge all members of Project SOAR’s research team including the US-based and country-based investigators and their staff for integrating research-utilization activities into their research. Finally, we would like to thank all the stakeholders in the countries from community to national level who participated and engaged in Project SOAR’s research-utilization activities.

Funding

This work was supported by Project SOAR (Cooperative agreement AID-OAA-A-14-00060), made possible by the generous support of the American people through the United States President’s Emergency Plan for AIDS Relief (PEPFAR) and United States Agency for International Development (USAID). The contents of this paper are the sole responsibility of the authors and do not necessarily reflect the views of PEPFAR, USAID, or the United States Government.

The Population Council leads Project SOAR in collaboration with Avenir Health, Elizabeth Glaser Pediatric AIDS Foundation, Johns Hopkins University, Palladium, and The University of North Carolina at Chapel Hill.

Author information

Affiliations

Authors

Contributions

SK developed the manuscript and TN and IS provided extensive reviews of the various drafts of the manuscript. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Samuel Kalibala.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

The three authors (SK, TN and IS) have consented for the publication of this manuscript.

Competing interests

Two of the authors (SK and TN) were employed by Palladium Group that was a member of the consortium of organizations under Project SOAR. The other author (IS) is an employee of Palladium Group but was not part of Project SOAR. The authors declare no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kalibala, S., Sinai, I. & Nutley, T. Documenting HIV research-utilization activities, outputs and outcomes: examples and lessons learned from Project SOAR. Arch Public Health 79, 99 (2021). https://doi.org/10.1186/s13690-021-00628-x

Download citation

Keywords

  • Research utilization
  • Logic model
  • Implementation science
  • HIV/AIDS
  • Measuring research outcomes