Instructions: This document provides an overview of the different common data collection methods and shows both the advantages and limitations of each. Before conducting an evaluation, a close review of the document is recommended to consider the type and specific framework for the evaluation.
| Method | Description | Advantages | Limitations |
|---|---|---|---|
|
Desk review of existing reports and documents |
Existing documentation, including quantitative and qualitative information about the project and its outputs and outcomes, such as documentation from capacity development activities, donor reports, digital records and other evidence. |
The information exists and is accessible at a low cost. |
May be time-consuming to put together and analyse. Evidence can be difficult to codify and analyse. Difficult to verify reliability and validity of data. |
|
Questionnaires |
Provide a standardized approach to obtaining information on a wide range of topics from a large number or diversity of stakeholders (usually employing sampling techniques) to obtain information on their attitudes, beliefs, opinions, perceptions, level of satisfaction, and so forth, concerning the operations, inputs, outputs and contextual factors of a project. |
Good for quickly gathering descriptive data on a wide range of topics at a relatively low cost. May be easier to analyse. |
May lead to bias, such as social desirability bias. May provide a general picture but may lack depth. May provide information out of context. Data may be subject to sampling bias. For online surveys, the number of respondents may not be controlled. |
|
In-depth interviews |
Solicit person-to-person responses to questions designed to obtain in-depth information about a person’s impressions or experiences, or to learn more about his or her answers to questionnaires or surveys. |
Facilitates fuller coverage, range and depth of information on a topic. |
Can be difficult to analyse. Potential for interviewer to bias against participant’s responses. |
| Method | Description | Advantages | Limitations |
|---|---|---|---|
|
On-site observation |
Entails use of observation form to record accurate information on site about how a project operates. |
Can see operations of a project as they occur. Can adapt to events as they occur. |
Can be difficult to interpret observed behaviours. Subject to site selection bias. |
|
Focus groups |
Small group (6 to 12 people) discussion to explore stakeholder opinions and judgements towards an activity, process, project or policy. They can also be used to collect in-depth information on the needs, motivations, intentions and experiences of the group. |
Useful to obtain in-depth qualitative information. |
Requires qualified facilitator. Can be difficult to analyse and interpret. Subject to facilitator bias. |
|
Key informants |
Qualitative in-depth interviews, often one-onone, with a wide range of stakeholders who have firsthand knowledge about the initiative’s operations and context. These community experts can provide particular knowledge and understanding of problems and recommend solutions. |
Can provide insight on the nature of problems and recommend solutions. Can provide different perspectives on a single issue or on several issues. |
Subject to sampling bias. Must have some means to verify or corroborate information. |
|
Case studies |
Involves comprehensive examination of cases to obtain in-depth information with the goal to fully understand the operational dynamics, activities, outputs, outcomes and interactions of a development project. |
Useful to fully explore factors that contribute to outputs and outcomes. |
Requires considerable time and resources not usually available for commissioned evaluations. Can be difficult to analyse and not necessarily replicable. |
This form is designed to determine to what extent the reintegration process of a migrant has been sustainable, that is to what extent a condition has been achieved “where returnees have reached a level of economic self-sufficiency, social stability within their community and psychosocial well-being that enables them to cope with (re)migration drivers”.97 Aiming to cover all aspects of this definition of sustainability, the questions below cover the three different dimensions of reintegration: economic, social and psychosocial. Without prejudice to the importance of the assistance provided by IOM through AVRR/PARA programmes, the main focus is not to assess the satisfaction of the migrant with IOM’s assistance throughout the reintegration process, but to evaluate to what extent the migrant’s reintegration has been sustainable.
The survey, which has been kept as short as possible, enables IOM staff to generate a composite (overall) reintegration score, as well as separate scores for the sustainability of reintegration in the economic, social and psychosocial dimensions. The example below of three respondents from IOM’s MEASURE project illustrates how reintegration experiences vary widely. The scoring system presents an opportunity to understand individual reintegration needs with a new level of insight.
On an individual level, these scores can be easily visualized similarly to the displays above. These images show the programmatic value of having individual dimensional scores: for example, while the 44-year old Ethiopian returnee needs significant assistance across all dimensions, the scores show that he is particularly vulnerable in the economic dimension. Similarly, while the 19-year old Afghan returnee is very well reintegrated overall, interventions should primarily focus on his psychosocial needs. Finally, the 24-year old Iraqi returnee appears to be better reintegrated in the social and psychosocial dimensions but requires a more intensive approach to her economic reintegration.
The scoring system, as well as interpretation of resulting scores, is further explained in a methodological note on scoring reintegration sustainability. For a copy of the methodological note, please contact: MPAHQTeam@iom.int. The methodological note also offers further guidance on the use of the survey, such as timing. IOM staff are advised to familiarize themselves with the methodological note before proceeding to study the survey itself as outlined below.
This form should be completed by staff during a structured interview with the returnee. The survey can serve as a baseline and progress assessment before and during the period of reintegration assistance and for final evaluation of returnee sustainability after the provision of reintegration assistance was concluded, as outlined below:
It is recommended that this survey is administered in person by a staff member who is or was not directly responsible for the provision of reintegration assistance to the beneficiary. This increases the likelihood that respondents will express their feelings without hesitation, avoiding what is termed “social desirability bias.”
This document offers a closer look at the indicators and questions used, guiding staff through the exact interpretation of each indicator. Page 2 contains the survey protocol, Pages 3–9 contain the survey template, annotated to facilitate understanding of the indicators.
SURVEY PROTOCOL
The survey should be conducted in a private space where returnees may feel comfortable reflecting on their experience and answering potentially sensitive questions. They should never be forced to answer any question and have the right to interrupt the interview at any time.
Protocol:
-
Prior to meeting, the staff member completes Profile and Section 1 of the survey. Information should be verified with the beneficiary, and any outstanding questions from these sections answered. Categories “selected” in Section 1 determine the composition of the survey questionnaire later administered to each returnee. (Staff only ask questions in sections “selected” in Section 1.)
-
Interviewer reads prompt to beneficiary and seeks their consent.98 If obtained, interviewer proceeds to Section 2.
-
For all questions: Interviewer reads questions out loud.
-
Interviewer observes instructions below each question:
“prompt” indicates that the interviewer should read answer options, and allow respondent to select the most appropriate.
“do not prompt” indicates that the interviewer should not read a list of possible answers to the respondent. Instead, interviewer should listen to the respondent’s free response, and select answer(s) closest to their own words.
“select one” indicates that the question can only have one answer.
“select all applicable” indicates that the question can have multiple answers.
-
Interviewer records answers and notes.
-
If selected answers refer to follow-up questions (such as in Question no. 9), Interviewer proceeds to follow up question (marked by question number in brackets, for instance “(10)”).
REINTEGRATION SUSTAINABILITY SURVEY
| Reintegration sustainability | |||
|---|---|---|---|
|
ECONOMIC DIMENSION Questions 1–10 contain indicators of economic reintegration, which contribute to economic self-sufficiency |
|||
| Questions | Answers | Notes | |
|---|---|---|---|
| 1 |
How satisfied are you with your current economic situation? select one |
|
For staff needs, and follow-up explanations. Anything in this column is not used for score calculation, but could be useful for case management purposes. |
| 2 |
Since you returned, how often have you had to reduce the quantity or quality of food you eat because of its cost? select one |
|
Given that this indicator is cross-sectional (has implications also for social and psychosocial dimensions of reintegration), it is weighted more heavily in the scoring system to reflect its overall importance in determining sustainability of reintegration. More information is available in the methodological note. |
| 3 |
Are you able to borrow money if you need to? select one |
|
|
| 4 |
Do you borrow money? How frequently? select one |
|
| Questions | Answers | Notes | |
|---|---|---|---|
| 5 |
On average, which amount is bigger: your spending every month, or your debt? select one |
|
|
| 6 |
How would you rate your access to opportunities (employment and training)? select one |
|
|
| 7 |
Do you currently work? select one |
|
|
| 8 |
Do you own any of the following productive assets? select all applicable |
|
| Questions | Answers | Notes | |
|---|---|---|---|
| 9 |
Are you currently looking for a job? select one |
|
If respondent indicates YES as an answer, please do include Q10. If respondent indicates NO or I DON’T WISH TO ANSWER, please skip Q10, and continue to Q11. |
| 10 |
Why are you looking for a new job? only if “yes” selected aboveselect all applicable |
|
|
|
SOCIAL DIMENSION Questions 11–21 contain indicators of social reintegration, reflecting the extent to which returnees have reached social stability within their community, including access to services relating to housing, education, justice, health and other public infrastructure services. |
|||
| 11 |
How would you rate your access to housing in your community? select one |
|
|
| 12 |
How would you rate the standard of housing you live in today? select one |
|
|
| Questions | Answers | Notes | |
|---|---|---|---|
| 13 |
How would you rate the access to education in your community? select one |
|
|
| 14 |
Are all school-aged children in your household currently attending school? select onedo not prompt |
|
|
| 15 |
How would you rate the access to justice and law enforcement in your community? select one |
|
|
| 16 |
Do you have at least one identification document? select one |
|
|
| 17 |
How would you rate the access to documentation (personal ID, birth certificates and so on) in your community? select one |
|
| Questions | Answers | Notes | |
|---|---|---|---|
| 18 |
How would you rate the access to safe drinking water in your community? select one |
|
|
| 19 |
How would you rate the access to health care in your community? select onedo not prompt |
|
Please explain why health care is not easily accessible to you:
|
| 20 |
What is the quality of health care available to you? select one |
|
|
| 21 |
Access to public services overall is generated from average answers to above questions (Q13, 15, 17, 18, 19). |
||
|
PSYCHOSOCIAL DIMENSION Questions 22–32 contain indicators of psychosocial reintegration, encompassing the emotional and psychological elements of reintegration. |
|||
| 22 |
How often are you invited or do you participate in social activities (celebrations, weddings, other events) within your community? select one |
|
|
| Questions | Answers | Notes | |
|---|---|---|---|
| 23 |
How do you feel about your support network? Can you rely on the network’s support? select one |
|
|
| 24 |
Do you feel you are part of the community where you currently live? select one |
|
|
| 25 |
How physically safe do you feel for yourself and your family during everyday activities outside? select one |
|
Given that this indicator is cross-sectional (has implications also for social and economic dimensions of reintegration), it is weighted more heavily in the scoring system to reflect its overall importance in determining sustainability of reintegration. |
| 26 |
How frequently have you experienced important tensions or conflicts between you and your family since you returned? select one |
|
For case management follow up: do you experience more incidents of tension than before your migration experience? |
| Questions | Answers | Notes | |
|---|---|---|---|
| 27 |
Have you felt discriminated against since your return? select one |
|
Follow up: if yes, please explain. |
| 28 |
Do you often suffer from any of the following? prompt |
|
|
| 29 |
Would you wish to receive specialized psychological support? select one |
|
|
| 30 |
Do you feel that you are able to stay and live in this country? select one |
|
Given that this indicator is cross-sectional (has implications also for social and economic dimensions of reintegration), it is weighted more heavily in the scoring system to reflect its overall importance in determining sustainability of reintegration. |
| Questions | Answers | Notes | |
|---|---|---|---|
| 31 |
What is it that makes you feel that way? only if “no” answered above |
|
|
| 32 |
Who are the people and organizations that support you in this community? select all applicable |
|
97 For IOM definition of sustainable reintegration, see Towards an Integrated Approach to Reintegration in the Context of Return (IOM, 2017).
98 When conducted in person, consent should be written. When interview conducted by phone, explicit, beneficiary should be asked to give explicit, informed consent verbally.
99 This survey can be taken repeatedly to show progress in reintegration sustainability following migrants’ return. Please refer to methodological note for further information.
TITLE [MIDTERM/FINAL/OTHER EVALUATION FOR “PROJECT”]
Commissioned by: Specify who is commissioning the evaluation report.
Evaluation context
Write a few paragraphs about the context of the evaluation. A few paragraphs about the project(s) that is to be evaluated and a general description of the relevant political, environmental, social, economic and legal context is usually sufficient.
Evaluation purpose
In this section, briefly explain why the evaluation is being conducted and why it is being conducted at this time. Specify the intended audience for the evaluation and how the evaluation will be used.
Some examples of audience and purpose are as follows:
-
The evaluation is being conducted for use by management, so that they can improve the implementation of an ongoing set of activities, projects or programmes.
-
The evaluation is being conducted for use by stakeholders, so that they can assess the relevance and accountability of a project for intended beneficiaries.
-
The evaluation is being conducted for use by the project team, so that they can document lessons learned and best practices from a completed set of activities.
-
The evaluation is being conducted for use by a donor, so that they can assess value for money for a set of activities that they have funded.
-
The evaluation is being conducted for use by senior management, so they can assess organizational effectiveness in implementing a strategy.
It is fairly common for an evaluation to be intended for use by a variety of audiences, such as project management, senior management, stakeholders and donors. If this is the case, briefly describe all of the evaluation’s main intended audiences and uses. Keep in mind the principle of intentionality in evaluations, which means that evaluations should only be undertaken if there is a clear intention to use the evaluation findings (refer to UNEG Norms for Evaluation in the UN system).
Evaluation scope
Briefly describe what the evaluation will cover and will not cover. This should include the time period to be covered (that is, the intervention period being evaluated, not the period of time available to complete the evaluation), the phases of a project to be covered and the geographical area to be covered. If there is a specific project, state its name. If there are specific exclusions – for example, if a project is being implemented in six provinces but two are inaccessible and will not be included in the evaluation – state them clearly.
Make sure that the evaluation scope is sufficient to achieve the evaluation purpose. For example, if the purpose is to assess value for money, but only the first three months of project implementation are being evaluated, the evaluation is unlikely to be able to achieve its purpose. Similarly, ensure that the scope of the evaluation is feasible within time and resource restraints.
Evaluation criteria
Specifically list the evaluation criteria that will form the basis of the evaluation.
Evaluation questions
For each of the listed criteria, specify the evaluation questions that the evaluator will answer. Cluster them according to the criteria. These questions should be specifically tailored to the needs of this evaluation.
Evaluation methodology
In this section, describe the data collection and analysis methods that will be used to conduct the evaluation. Refer to Annex 4.A for a description of different data collection and analysis methods. Indicate how the evaluation will address relevant cross-cutting themes of the rights-based approach to programming, gender mainstreaming, environmental sensitivity and sustainability, sustainability of results, principled humanitarian action and mainstreaming protection into crisis response.
Bear in mind that it might be necessary for this section to be more general in nature, pending development of a more detailed methodology following discussions with the selected evaluator or evaluation team. This is particularly the case when the evaluation manager lacks technical expertise and intends to solicit the advice of the evaluator on the most appropriate methodologies for the evaluation.
Finally, state that the evaluation must follow UNEG norms and standards for evaluations, and relevant ethical guidelines.
Evaluation deliverables
List the deliverables the evaluator will be responsible for providing. This usually includes an inception report, a presentation outlining the initial findings and a final report.
Evaluation workplan
In this section, describe the following:
-
The activities to be conducted and the amount of time (how many days, weeks or months) that will be allocated for completing each activity.
-
The roles and responsibilities of each member of the evaluation team and of the stakeholders.
-
The processes for quality assurance. At a minimum, this should include: (a) the agreement on the final terms of reference between the evaluation manager and the evaluator or evaluation team; (b) review, revision and acceptance of the inception report; (c) review, revision and acceptance of the final report. It is also standard practice to have a management meeting at the beginning of the evaluation process to ensure that the evaluation manager, the evaluator or evaluation team, and stakeholders (if relevant) all share a common understanding of the evaluation process and various roles and responsibilities, as well as to have a debrief and presentation of initial findings following conclusion of the data collection and preliminary analysis. This allows for any obvious oversights, misinterpretations or information gaps to be identified and addressed before the evaluator begins drafting the final report.
This information can be provided either in narrative text or in the table below. If using both the narrative text and the table, review the information carefully to ensure that what is written in the narrative matches with what is written in the table.
| Activity | Days | Responsible | Location | Days/Weeks/Months | |||||
|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | ||||
| Insert individual activities to be conducted during the evaluation. | Indicate how many days are needed for each activity. | Specify who is responsible for completing the activity | Specify where the activity is to be conducted. | ||||||
| Example: Review project documents and relevant literature. | 3 | Evaluator | Home based | ||||||
Evaluation budget
Inclusion of this section is at the discretion of the evaluation manager. In some contexts, it is appropriate to simply specify the total budget available for the evaluation or to provide a more detailed budget (such as the amount to be paid upon receipt and acceptance of each deliverable or to specify the amount available for fees, travel, daily subsistence allowance, equipment, data collection and others). In other contexts, it may be preferable to not specify the budget and instead have applicants propose a budget in their applications.
Instructions: This checklist provides guidance on the different steps to be undertaken during an evaluation and at what stage; confirmation that no crucial step has been forgotten is vital for the evaluation.
| Preparation of the Evaluation | |
|---|---|
|
The overall objective and purpose of the evaluation has been defined (analysis of AVRR programme’s performance and accountability, exploration of new modalities for implementation and so on). |
|
|
The focus and scope of the evaluation has been defined (focus is mainly related to evaluation criteria of relevance, effectiveness, cost-benefit, efficiency, outcome, sustainability and long-term impact of the AVRR programme). |
|
|
A decision has been taken as to whether the evaluation will be carried out by an internal or external evaluator and budget provision guaranteed. |
|
|
The methods for data collection have been defined (review of existing documents and report, questionnaires, in-depth interviews, on-site evaluation, focus groups, key informants and case studies) in line with the timing and resources available for the evaluation. |
|
|
The Terms of Reference for the evaluation have been drafted, having considered the following elements below: - The Background section summarizes the context of the project that will be evaluated. The expected outcomes and outputs of the projects are stated as they will be one of the main references of the evaluation (to list indicators could be too detailed unless only a few indicators were listed in the initial project document). - The objective(s) of the evaluation specify the ‘why do it’, the nature of the evaluation to be undertaken and the product it is meant to deliver, the intended audience, the use of the evaluation and the involvement of the stakeholders in the evaluation. - The methodology section covers the approach for data collection and data analyses in a precise manner, ensuring that the choice for the duration as well as the techniques to be applied during the evaluation adequately reflect the available budget (taking into account potentially high costs in the event that a large number of interviews are carried out with returnees in different countries of origin). - The role of the various parties involved in the evaluation (IOM, project partners, beneficiaries and, if included, steering committees) is clearly defined, enabling all parties to know what they are responsible for and what is expected from them, such as providing information on the management of the project, allowing access to project–related documentation and collecting data from the government. - The budget lays out (if possible, in detail) the resources required to conduct the evaluation, including potential consultancy fees and costs of data collection and surveys; the resources in kind (such as transportation or administrative support) which will be made available for the evaluation team, are clearly reflected. - The team composition is described (a single evaluator or a team with different expertise and skills). - Deliverables that will be generated at various stages of the evaluation process (such as work plan, inception report, mid-term report, final report and recommendations) are included. |
|
|
- The schedule sets out in chronological order the dates by which certain activities have to be completed. This includes a consideration of possible risks that might have an impact on the timing of the evaluation (such as being unable to contact migrants for monitoring purposes upon return). - Relevant cross-cutting aspects, such as gender and human rights are duly considered in the ToRs and in the evaluation as a whole. - Data protection principles are embedded in the evaluation’s methodology. - An ethical framework is established for the inclusion of vulnerable groups. - Adherence to UNEG Norms, Standards and Ethical Guidelines for Evaluation and Evaluators.100
|
|
| Managing and Implementing Evaluation | |
|---|---|
|
The evaluation consultant or team has been selected, based on the following considerations: - The evaluator(s) has the appropriate educational background and training for the evaluation (social sciences, specialized training in evaluation, project management, social statistics or statistical research and analysis, specific expertise such as economics or microcredits, all depending on the nature of the evaluation). - The evaluator(s) has sufficient background and experience with AVRR or IOM or UN evaluations in general, and with the different methodologies identified for data collection in particular, - The evaluator(s) has sufficient knowledge about the other areas to be evaluated (for instance AVRR polices and legislation) as well as of the local context (host vs. origin country, social and economic situations, security and stabilization policies) in which the evaluation is taking place. - The evaluator(s) has appropriate oral and written communication skills. - If the evaluator(s) is given access to confidential information, a confidentiality agreement has been signed with them. |
|
|
A preparatory workshop has been carried out to discuss relevant aspects of the evaluation, such as clarifying the roles and coordination of the various stakeholders (in particular when adopting participatory approaches). |
|
|
The project evaluator(s) has been introduced to the AVRR project team and other relevant stakeholders, and is briefed about the nature and objective of the evaluation. |
|
|
Assistance to the evaluator(s) is provided by the AVRR project team throughout the process of data collection as needed (such as by arranging interviews with migrants and other actors, identifying respondents for questionnaires, organizing site visits to returnees’ places of work or meetings). |
|
|
Follow-up meetings are organized between the reintegration team and the evaluator(s) to monitor the work and provide input, if needed, respecting the independence of the evaluator |
|
|
If foreseen in the ToRs, the inception report and/or an interim report is shared with the AVRR project team or Chief of Mission or relevant stakeholders for their inputs. |
|
|
The final evaluation report responds clearly to the objectives of the evaluation, is logically structured and contains evidence-based findings, conclusions, recommendations and lessons. |
|
|
The project team is given the opportunity to provide input with regards to the content, structure, and length of the report, keeping in mind the independence of the evaluators regarding the actual findings and recommendations of the report. |
|
|
A quality review of the final evaluation report is conducted prior to publication, including a revision as to whether the report addresses the objectives of the evaluation, that it has been well prepared and is clearly presented.101 |
|
|
A review of the findings and recommendations of the final report takes place in coordination with relevant stakeholders. |
|
|
A debriefing (such as a workshop or conference) is organized for the donor, the national government, partners and other stakeholders regarding the results of the evaluation as well as possible follow up. The report is equally made available to other offices, Headquarters and partners for future sharing of best practices. |
|
|
Concrete actions for follow-up on implementation of the recommendations are discussed with the actors for whom the evaluation was conducted. |
|
100 See Norms and Standards for Evaluation (UNEG, 2016).
101 Quality review checklists for Evaluation ToRs and Evaluation Reports are available at the IOM Evaluation Webpage/technical references.