Assignment: Designing a Process Evaluation
What is process evaluation?
Process evaluation uses empirical data to assess the delivery of programs. In contrast to outcome evaluation, which assess the impact of the program, process evaluation verifies what the program is and whether it is being implemented as designed. Thus, process evaluation asks “what,” and outcome evaluation asks, “so what?”
When conducting a process evaluation, keep in mind these three questions:
1. What is the program intended to be? 2. What is delivered, in reality? 3. Where are the gaps between program design and delivery?
This workbook will serve as a guide for designing your own process evaluation for a program of your choosing. There are many steps involved in the implementation of a process evaluation, and this workbook will attempt to direct you through some of the main stages. It will be helpful to think of a delivery service program that you can use as your example as you complete these activities. Why is process evaluation important? 1. To determine the extent to which the program is being implemented according to plan 2. To assess and document the degree of fidelity and variability in program implementation, expected or unexpected, planned or unplanned
3. To compare multiple sites with respect to fidelity 4. To provide validity for the relationship between the intervention and the outcomes 5. To provide information on what components of the intervention are responsible for outcomes 6. To understand the relationship between program context (i.e., setting characteristics) and program processes (i.e., levels of implementation).
7. To provide managers feedback on the quality of implementation 8. To refine delivery components 9. To provide program accountability to sponsors, the public, clients, and funders 10. To improve the quality of the program, as the act of evaluating is an intervention.
Evaluation Expert Session July 16, 2002 Page 2
Stages of Process Evaluation Page Number
1. Form Collaborative Relationships 3 2. Determine Program Components 4 3. Develop Logic Model* 4. Determine Evaluation Questions 6 5. Determine Methodology 11 6. Consider a Management Information System 25 7. Implement Data Collection and Analysis 28 8. Write Report** Designing a Process Evaluation
Also included in this workbook:
a. Logic Model Template 30 b. Pitfalls to avoid 30 c. References 31
Evaluation can be an exciting, challenging, and fun experience
* Previously covered in Evaluation Planning Workshops. ** Will not be covered in this expert session. Please refer to the Evaluation Framework
and Evaluation Module of FHB Best Practice Manual for more details.
Evaluation Expert Session July 16, 2002 Page 3
Forming collaborative relationships
A strong, collaborative relationship with program delivery staff and management will likely result in the following:
Feedback regarding evaluation design and implementation Ease in conducting the evaluation due to increased cooperation Participation in interviews, panel discussion, meetings, etc. Increased utilization of findings
Seek to establish a mutually respectful relationship characterized by trust, commitment, and flexibility.
Key points in establishing a collaborative relationship:
Start early. Introduce yourself and the evaluation team to as many delivery staff and management personnel as early as possible.
Emphasize that THEY are the experts, and you will be utilizing their knowledge and information to inform your evaluation development and implementation.
Be respectful of their time both in-person and on the telephone. Set up meeting places that are geographically accessible to all parties involved in the evaluation process.
Remain aware that, even if they have requested the evaluation, it may often appear as an intrusion upon their daily activities. Attempt to be as unobtrusive as possible and request their feedback regarding appropriate times for on-site data collection.
Involve key policy makers, managers, and staff in a series of meetings throughout the evaluation process. The evaluation should be driven by the questions that are of greatest interest to the stakeholders. Set agendas for meetings and provide an overview of the goals of the meeting before beginning. Obtain their feedback and provide them with updates regarding the evaluation process. You may wish to obtained structured feedback. Sample feedback forms are throughout the workbook.
Provide feedback regarding evaluation findings to the key policy makers, managers, and staff when and as appropriate. Use visual aids and handouts. Tabulate and summarize information. Make it as interesting as possible.
Consider establishing a resource or expert “panel” or advisory board that is an official group of people willing to be contacted when you need feedback or have questions.
Designing a Process Evaluation
Evaluation Expert Session July 16, 2002 Page 4
Determining Program Components
Program components are identified by answering the questions who, what, when, where, and how as they pertain to your program.
Who: the program clients/recipients and staff What: activities, behaviors, materials When: frequency and length of the contact or intervention Where: the community context and physical setting How: strategies for operating the program or intervention
BRIEF EXAMPLE: Who: elementary school students What: fire safety intervention When: 2 times per year Where: in students’ classroom How: group administered intervention, small group practice
1. Instruct students what to do in case of fire (stop, drop and roll). 2. Educate students on calling 911 and have them practice on play telephones. 3. Educate students on how to pull a fire alarm, how to test a home fire alarm and how to change batteries in a home fire alarm. Have students practice each of these activities. 4. Provide students with written information and have them take it home to share with their
parents. Request parental signature to indicate compliance and target a 75% return rate. Points to keep in mind when determining program components Specify activities as behaviors that can be observed
If you have a logic model, use the “activities” column as a starting point
Ensure that each component is separate and distinguishable from others
Include all activities and materials intended for use in the intervention
Identify the aspects of the intervention that may need to be adapted, and those that should always be delivered as designed. Consult with program staff, mission statements, and program materials as needed.
Designing a Process Evaluation
Evaluation Expert Session July 16, 2002 Page 5
Your Program Components
After you have identified your program components, create a logic model that graphically portrays the link between program components and outcomes expected from these components.
Now, write out a succinct list of the components of your program. WHO: WHAT: WHEN: WHERE: HOW:
Evaluation Expert Session July 16, 2002 Page 6
What is a Logic Model
A logical series of statements that link the problems your program is attempting to address (conditions), how it will address them (activities), and what are the expected results (immediate and intermediate outcomes, long-term goals).
Benefits of the logic model include:
helps develop clarity about a project or program, helps to develop consensus among people, helps to identify gaps or redundancies in a plan, helps to identify core hypothesis, helps to succinctly communicate what your project or program is about.
When do you use a logic model Use… – During any work to clarify what is being done, why, and with what intended results – During project or program planning to make sure that the project or program is logical and complete – During evaluation planning to focus the evaluation – During project or program implementation as a template for comparing to the actual program and as a filter to determine whether proposed changes fit or not. This information was extracted from the Logic Models: A Multi-Purpose Tool materials developed by Wellsys Corporation for the Evaluation Planning Workshop Training. Please see the Evaluation Planning Workshop materials for more information. Appendix A has a sample template of the tabular format. Designing a Process Evaluation.
Evaluation Expert Session July 16, 2002 Page 7
Determining Evaluation Questions
As you design your process evaluation, consider what questions you would like to answer. It is only after your questions are specified that you can begin to develop your methodology. Considering the importance and purpose of each question is critical.
BROADLY…. What questions do you hope to answer? You may wish to turn the program components that you have just identified into questions assessing: Was the component completed as indicated? What were the strengths in implementation? What were the barriers or challenges in implementation? What were the apparent strengths and weaknesses of each step of the intervention? Did the recipient understand the intervention? Were resources available to sustain project activities? What were staff perceptions? What were community perceptions? What was the nature of the interaction between staff and clients?
These are examples. Check off what is applicable to you, and use the space below to write additional broad, overarching questions that you wish to answer.
Evaluation Expert Session July 16, 2002 Page 8
SPECIFICALLY … Now, make a list of all the specific questions you wish to answer, and organize your questions categorically. Your list of questions will likely be much longer than your list of program components. This step of developing your evaluation will inform your methodologies and instrument choice. Remember that you must collect information on what the program is intended to be and what it is in reality, so you may need to ask some questions in 2 formats. For example:
How many people are intended to complete this intervention per week?” How many actually go through the intervention during an average week?”
Consider what specific questions you have. The questions below are only examples! Some may not be appropriate for your evaluation, and you will most likely need to add additional questions. Check off the questions that are applicable to you, and add your own questions in the space provided. WHO (regarding client): Who is the target audience, client, or recipient? How many people have participated? How many people have dropped out? How many people have declined participation? What are the demographic characteristics of clients?
Race Ethnicity National Origin Age Gender Sexual Orientation Religion Marital Status Employment Income Sources Education Socio-Economic Status
What factors do the clients have in common? What risk factors do clients have? Who is eligible for participation? How are people referred to the program? How are the screened? How satisfied are the clients?
Evaluation Expert Session July 16, 2002 Page 9
WHO (Regarding staff): Who delivers the services? How are they hired? How supportive are staff and management of each other? What qualifications do staff have? How are staff trained? How congruent are staff and recipients with one another? What are staff demographics? (see client demographic list for specifics.)
YOUR QUESTIONS: WHAT: What happens during the intervention? What is being delivered? What are the methods of delivery for each service (e.g., one-on-one, group session, didactic instruction,
etc.) What are the standard operating procedures? What technologies are in use? What types of communication techniques are implemented? What type of organization delivers the program? How many years has the organization existed? How many years has the program been operating? What type of reputation does the agency have in the community? What about the program? What are the methods of service delivery? How is the intervention structured? How is confidentiality maintained?
YOUR QUESTIONS: WHEN: When is the intervention conducted? How frequently is the intervention conducted? At what intervals? At what time of day, week, month, year? What is the length and/or duration of each service?
Evaluation Expert Session July 16, 2002 Page 10
YOUR QUESTIONS: WHERE: Where does the intervention occur? What type of facility is used? What is the age and condition of the facility? In what part of town is the facility? Is it accessible to the target audience? Does public transportation access
the facility? Is parking available? Is child care provided on site?
YOUR QUESTIONS: WHY: Why are these activities or strategies implemented and why not others? Why has the intervention varied in ability to maintain interest? Why are clients not participating? Why is the intervention conducted at a certain time or at a certain frequency?
Evaluation Expert Session July 16, 2002 Page 11
Validating Your Evaluation Questions
Even though all of your questions may be interesting, it is important to narrow your list to questions that will be particularly helpful to the evaluation and that can be answered given your specific resources, staff, and time.
Go through each of your questions and consider it with respect to the questions below, which may be helpful in streamlining your final list of questions. Revise your worksheet/list of questions until you can answer “yes” to all of these questions. If you cannot answer “yes” to your question, consider omitting the question from your evaluation.
Will I use the data that will stem from these questions?
Do I know why each question is important and /or valuable?
Is someone interested in each of these questions?
Have I ensured that no questions are omitted that may be important to someone else?
Is the wording of each question sufficiently clear and unambiguous?
Do I have a hypothesis about what the “correct” answer will be for each question?
Is each question specific without inappropriately limiting the scope of the evaluation or probing for a specific response?
Do they constitute a sufficient set of questions to achieve the purpose(s) of the evaluation?
Is it feasible to answer the question, given what I know about the resources for evaluation?
Is each question worth the expense of answering it?
Derived from “A Design Manual” Checklist, page 51.
Evaluation Expert Session July 16, 2002 Page 12
Determining Methodology Process evaluation is characterized by collection of data primarily through two formats: 1) Quantitative, archival, recorded data that may be managed by an computerized
tracking or management system, and 2) Qualitative data that may be obtained through a variety of formats, such as surveys or focus groups.
When considering what methods to use, it is critical to have a thorough understanding and knowledge of the questions you want answered. Your questions will inform your choice of methods. After this section on types of methodologies, you will complete an exercise in which you consider what method of data collection is most appropriate for each question. Designing a Process Evaluation.
Do you have a thorough understanding of your questions?
Furthermore, it is essential to consider what data the organization you are evaluating already has. Data may exist in the form of an existing computerized management information system, records, or a tracking system of some other sort. Using this data may provide the best reflection of what is “going on,” and it will also save you time, money, and energy because you will not have to devise your own data collection method! However, keep in mind that you may have to adapt this data to meet your own needs – you may need to add or replace fields, records, or variables.\
Click here to ORDER an A++ paper from our MASTERS and DOCTORATE WRITERS:Assignment: Designing a Process Evaluation
What data does your organization already have? Will you need to adapt it?
If the organization does not already have existing data, consider devising a method for the organizational staff to collect their own data. This process will ultimately be helpful for them so that they can continue to self-evaluate, track their activities, and assess progress and change. It will be helpful for the evaluation process because, again, it will save you time, money, and energy that you can better devote towards other aspects of the evaluation. Management information systems will be described more fully in a later section of this workbook. Designing a Process Evaluation.
Do you have the capacity and resources to devise such a system? (You may need to refer to a later section of this workbook before answering.)
Evaluation Expert Session July 16, 2002 Page 13
In the world today, the burdens of mental health illness is rising among different countries revealing a trend that needs redress. In the United States of America alone, the National Institute for Mental Health (NIMH) indicates that depression is the most common form of mental health disorder with a prevalence of 9.1% among adults (Alterman et al., 2018). Report from researches from the World Health Organization (WHO) postulates that major depression is anticipated to be the second most leading cause of morbidity across the world by the year 2020. Besides, depression emerge as the second most common chronic illness reported in primary care and often affects the treatment of co-occurring conditions (Fathian et al., 2015). Currently, major depressive disorders cause disability in young adults between the age range of 15 and 44. However, male Hispanic immigrants suffer the most from major depression due to factors related to their minority status (Estrada-Martínez, Lee & Shapiro, 2019). If left untreated within the right time, depression can result into prolonged emotional suffering, substance misuse, adoption of risky behavioral practices and even suicide. The Predisposing, Reinforcing, and Enabling Constructs in Educational Diagnosis and Evaluation (PRECEDE) together with the Policy, Regulatory, and Organizational Constructs in Educational and Environmental Development (PROCEED) otherwise known as the Precede-Proceed Model (PPM) may help address depression (Hlaing, Sullivan & Chaiyawat, 2019). The PPM framework works on the premise to identify the predisposing factors of depression and providing a solution to major depressive disorders. However, the underlying concepts of using the precede-proceed model to identify factors predisposing Latino men to depression and to examine the effects related to these behaviors to influence well-being remains unclear.
The present study delves into a health promotion intervention for the Latino men affected by depression. The article uses the precede-proceed model as a framework to identify the factors that predispose the target group to major depressive disorders. From this analysis, PPM will be used as a planning tool to identify useful steps that can be implemented to provide intervention for the Latino men. Nonetheless, PPM will be adopted to address those who influence the behavior as well as the environmental factors related to depression to determine behavior change for the target population.
A Description of the Precede-Proceed Model
Precede-proceed framework is a comprehensive structure that is important in addressing health needs for a given target population. PPM is used to design, implement and for the evaluation of public health programs mainly to promote health outcomes for a specific population group (Hlaing, Sullivan & Chaiyawat, 2019). To start, the PRECEDE part of the framework entails conducting assessments to determine several community factors related to the health issue and it is defined by four phases. The first phase is social assessment performed in order to identify the expected outcomes. The second phase which is the epidemiological assessment is used as a platform to identify possible determinants of health for the target problem. This provides information on factors affecting the population to help stakeholder in setting goals and priorities for the intervention process.
The third phase of the PRECEDE is the ecological assessment whereby the behavioral together with the environmental factors that predispose or reinforce a particular lifestyle for the study population is identified (Hlaing, Sullivan & Chaiyawat, 2019). The premise assists in the implementation of the proposed intervention in a bid to promote desired behavior change among the target population. For phase four, administrative as well as policy factors that influence the proposed interventions are analyzed to promote and support the expected change in behavior for a given target population.
Conversely, the PROCEED part of the PPM framework relates to the implementation of the identified intervention to achieve expected outcomes. This section is as well made up of four phases which continue from the stages described above. With this, the implementation is labelled phase five and this refers to the development of the design necessary for the intervention process. This phase also involves an assessment of the availability of resources necessary for the final implementation of the selected program (Hlaing, Sullivan & Chaiyawat, 2019). Phase six of the model is the implementation of the program to achieve the desired goals for the selected target population. Phase seven of the PROCEDE model focuses on impact evaluation which involves the analysis of the behavior change for the selected population. Outcome evaluation as phase eight is the final phase of the PROCEDE framework and it assesses positive outcomes in the identified change model for the selected population. Nonetheless, the phase evaluates if there is an increase or a decrease in the prevalence of a specific undesirable behavior for the identified target population. The precede-proceed framework is described in the context of depression for Latino men in South Florida to provide an intervention to the target population. With this, the description starts from Phase 2 of the PRECEDE-PROCEED model as outlined below.
Using the Phase 2: Epidemiological, Behavioral, and Environmental Assessment
Hispanic-Americans are the fastest growing minority racial groups in the United States of America. The community makes up 17% of the American population and it is estimated to have 58 million people (Abraído-Lanza et al., 2016). The U.S. Census Bureau define a Hispanic to be a person of Mexican, Puerto Rican, Spanish, Cuban, South or Central American origin. According the reports by the Center for Disease Control and Prevention (CDC), there are high rates of mental health illness associated with Hispanics (Estrada-Martínez, Lee & Shapiro, 2019). Specifically, the report identifies depression and psychological distress as the major mental health conditions affecting the racial group. Furthermore, the report reveals that the prevalence rates of depression for Hispanic-Americas is at 4% compared to 3.1% of the non-Hispanic whites (Alterman et al., 2018). Besides, the latter are three times more likely to receive mental health treatment as opposed to the Hispanics. The disparity in mental health access has disproportionately affected the Latino men in America which has led to a high prevalence rate of the population when compared to female Hispanics (Niedhammer et al., 2016). Moreover, Latino men are less likely to seek early treatment of mental health as opposed to women.
Using the Phase 3: Educational and Ecological Assessment
There are various social determinants of depression among Latino men in South Florida. Income, education, present health status, acculturation and employment status influence the prevalence of depression among the target population. A majority of the male Hispanics are not born in the United States of America. Besides, most of the Latino men have lived in America for an average of 8.5 years. Abraído-Lanza et al. (2016) reports that depressive symptoms increases when the target population stays longer in a non-native country. This is attributed to loss of cultural norms, values as well as lack of family cohesion. Loss of social coherence is considered as a predisposing factor to depression. This also relates to the marital status of the target population which influence level of interaction for male Hispanics (Sepandi et al., 2018). Ideally, Latino men are also likely to be predisposed to reinforcing factor related to binge drinking. There are numerous liquor stores in South Florida and this promotes compulsive behavior which are linked to mental health (Estrada-Martínez, Lee & Shapiro, 2019). The enabling factor for depression for the target population is related to daily hustles and engagement of the population in the income-generating activities. The population resides in poor resource settings and are therefore likely to face challenges of unemployment and low incomes that may not be adequate to improve health seeking behavior (Vaeth, Caetano & Mills, 2016). These findings are likely to support the promotion of health to understand the major reasons for the high rates of depression in Latino men.
Using the Phase 4: Administration and Policy Assessment
The National Alliance on Mental Illness (NAMI) develop policies in the management of mental health issues including depression for the Latino Community. These mainly focus on prompt treatment of medical conditions that can predispose the target population to the depression (Vaeth, Caetano & Mills, 2016). The U.S. Preventive Services Task Force (USPSTF) requires adult patients to be screened by providers to determine clinical signs of depression (Estrada-Martínez, Lee & Shapiro, 2019). This government institution present in South Florida requires Latino men to seek care to assist in early diagnosis and treatment of depression to improve quality of life of the target population. On the other hand, the Department of Health and Human Services (HHS) provide funding to expand provision of mental health services among the Latino men (De Oliveira et al., 2017). This mainly focus on educating the population about the condition, associated symptoms, negative outcomes and strategies to address the mental health issue. Moreover, the establishment of the Substance Abuse and Mental Health Services Administration (SAMHSA) provide policies on reducing the rates of the misuse of drugs that can predispose the population to issues of mental health (Niedhammer et al., 2016). The agency also respond to the concerns of compulsive alcohol drinking which increases the chances of developing depression among the Latino men.
How Phase 5-8 of the Proceed Part Apply to Depression among Latino Men
In these phases of PPM, implementation followed by the evaluation of the intervention are conducted. For phase 5, planning of the intervention is designed based on the unique needs of the target population. Besides, a budget plan is formulated to provide the list of human and financial resources that will be needed for the intervention. In this case, physicians, nurses and mental counsellors are recruited to support the program (De Oliveira et al., 2017).
Phase 6 of the model focuses on intervention in which health education is administered to the Latino men diagnosed with depression. Nonetheless, medications are administered by the physician in collaboration with psychiatrists to relieve anxiety symptoms related to depression for the target group (Lazarevic, Crovetto & Shapiro, 2018). Rehabilitation programs will as well be organized to address the target population whose depressive state emerged from alcohol and substance abuse.
As for phase 7, the intervention process will be evaluated to assess its relevance to the target population. Specifically, a clustered randomized controlled trials (RCTs) with regard to the implementation model will be organized to assess the approach of the program to the population. Moreover, the outcomes related to the implementation process will be monitored at the baseline as well as at the post-implementation stage (Fathian et al., 2015). Subsequently, a six-month follow-up will be made to assess the impact of the health promotion on depression outcomes for the target population. The time has been selected as it provides adequate period for assessing behavior of Latino men based on the intervention administered.
In the final phase, outcome evaluation of the whole project will be undertaken. Particularly, a multilevel structural equation model (MSEM) will be implemented to assess how the proposed interventions respond to the social cognitive factors of behavior change for the target population (Sepandi et al., 2018). This relates to how the aforementioned intervention reduces anxiety for the target population. Moreover, the overall outcome of the intervention will be reflected by how the implementation method improve health and mental status of the Latino men.
Summary of Precede-Proceed Model and its Implication on Depression among Latino Men
The PRECEDE-PROCEED framework as described in the preceding discussion provides a bundle of evidence in addressing social behavior change among the target population. This means that PPM is based on a structure that can be readily used to support planning and implementation of health promotion or specific disease prevention program to improve quality of health for a given population. The framework has been implemented in a myriad of settings just as the same that relates to depression among Latino men (De Oliveira et al., 2017). Moreover, the discussion unravels how the intervention can be achieved by several stakeholders to improve behavior change among the target population. Specifically, PPM requires the participation of all players in mental health ranging from healthcare workers to family members and other social networks for the target population (Lazarevic, Crovetto & Shapiro, 2018). Based on this premise, the target population will develop insight about the benefit of the program especially by focusing on how the intervention address their mental health. This will results in improved outcomes for the Latino men with regard to their mental state.
Abraído-Lanza, A. F., Echeverría, S. E., & Flórez, K. R. (2016). Latino immigrants, acculturation, and health: Promising new directions in research. Annual review of public health, 37, 219-236. DOI: 10.1146/annurev-publhealth-032315-021545.
Alterman, T., Grzywacz, J. G., Muntaner, C., Shen, R., Gabbard, S., Georges, A., … & Carroll, D. J. (2018). Elevated depressive symptoms among hired crop workers in the United States: Variation by sociodemographic and employment characteristics. Journal of Rural Mental Health, 42(2), 69.
De Oliveira, G., Cianelli, R., Gattamorta, K., Kowalski, N., & Peragallo, N. (2017). Social determinants of depression among Hispanic women. Journal of the American Psychiatric Nurses Association, 23(1), 28-36. DOI: 10.1177/1078390316669230.
Estrada-Martínez, L. M., Lee, H., & Shapiro, E. (2019). Trajectories of depressive symptoms from adolescence to adulthood among multiple Latino subgroups. Journal of Latinx Psychology.
Fathian, N., Jafari, M., Heidari, Z. H., & Samouei, R. (2015). Identification of behavior change methods in social marketing for countering induced demand in health system. International Journal of Health System and Disaster Management, 3(5), 6. DOI: 10.4103/2347-9019.168570.
Hlaing, P. H., Sullivan, P. E., & Chaiyawat, P. (2019). Application of PRECEDE-PROCEED Planning Model in Transforming the Clinical Decision Making Behavior of Physical Therapists in Myanmar. Frontiers in public health, 7, 114. DOI: 10.3389/fpubh.2019.00114.
Lazarevic, V., Crovetto, F., & Shapiro, A. F. (2018). Challenges of Latino young men and women: Examining the role of gender in discrimination and mental health. Children and Youth Services Review, 94, 173-179.
Niedhammer, I., Lesuffleur, T., Coutrot, T., & Chastang, J. F. (2016). Contribution of working conditions to occupational inequalities in depressive symptoms: results from the national French SUMER survey. International archives of occupational and environmental health, 89(6), 1025-1037.
Sepandi, M., Taghdir, M., Akarzadeh, I., & Alimohamadi, Y. (2018). Assessing the Effective factors on depression in Khuzestan women. Journal of Biostatistics and Epidemiology, 4(4), 252-258.
Vaeth, P. A., Caetano, R., & Mills, B. A. (2016). Factors associated with depression among Mexican Americans living in US–Mexico border and non-border areas. Journal of immigrant and minority health, 18(4), 718-727.