ALNAP: posted Guide to Real-time evaluations of humanitarian action

This pilot guide is intended to help both evaluation managers and team leaders in commissioning, overseeing and conducting real-time evaluations of humanitarian operational responses. Drawing on a synthesis of existing good practices, it is intended as a flexible resource that can be adapted to a variety of contexts.

This guide concentrates on RTEs undertaken in first phase of an emergency response – where the RTE fieldwork takes place within a few months of the start of the response. This is because such RTEs pose particular problems for both the evaluation manager and the evaluation team. RTEs that take place later on in the response are closer to ex-post humanitarian evaluations, but this guide also addresses how such RTEs can feed into ongoing operations.

The focus of this guide is therefore on what is distinctive about humanitarian RTEs. It does not offer advice on evaluation methodologies in general, but on specific aspects of methodology which make RTEs unique and different. Nevertheless some of the advice will apply to all evaluations and not just to RTEs. This is motivated partly by the authors’ observations of areas of weakness in existing evaluations.

Download the guide on the ALNAP website.

Updated: ALNAP State of the System report is out

Bringing together much of the work done by ALNAP since the tsunami, this first pilot report provides a baseline and working methodology which will be built upon and improved in subsequent iterations.

Download the report from the ALNAP website. Although there is an excellent executive summary, a quick run-down of the key points follows in the full post.

Aims of the report: Provide a system-level mapping and assessment of international humanitarian assistance over the last two years according to key criteria. Includes new, previously unavailable descriptive statistics and highlights some new initiatives in policy and practice.

Areas of focus: Operational performance of the ‘formal international humanitarian system’, focussing on emergencies involving international agencies and an appeal for international assistance. Limitations: did not eventually cover national, local and community-based organisations and does not measure beneficiary-level impacts.
Key findings:

• the international system has grown in staff size by an average annual rate of 6% over the past decade, and has reached a population of roughly 211,000 humanitarian workers in the field.
• In 2008, some $6.6 billion was contributed by donors directly to international emergency response efforts, and the combined humanitarian expenditures of aid organisations on overseas programme activities totalled around $12.8 billion.
• In terms of performance, findings indicate an overall positive trend in areas having to do with the internal workings of the humanitarian system – such as coordination mechanisms, funding vehicles and needs assessment tools – while at the same time some fundamental issues, such as leadership and the system’s engagement with and accountability to beneficiaries, remained weak. The findings thus depict a system steadily and incrementally improving its own internal mechanics and technical performance, while remaining deficient in some big picture requirements for effectiveness.

Findings against the OECD DAC review assessment criteria:

Coverage/sufficiency: still insufficient as despite growing sector and rise in funding, needs have also gone up. Nonetheless coverage is improving over time - over 85% of total stated requirements met in 2007 and 2008, compared with 81% in 2006 and only 67% in 2005. Declining in some contexts due to insecurity or host government restrictions. In the most contested environments, insecurity for aid workers has increased markedly.

Relevance/appropriateness: Quality of needs assessments still seen as a weakness, but have improved with a majority of respondents reporting adequate inter-agency needs assessments in their contexts and wider breadth of types of programming improving flexibility. However evaluations and beneficiary consultations show common instances of multiple assessments without sufficient follow-up. Beneficiaries continue to be inadequately consulted and involved in assessments and subsequent programme design. Prioritisation is improved but may be proliferating with too many parallel processes appearing.

Effectiveness: Responses are more timely thanks to significant agency investment in standby capacity and new mechanisms (e.g CERF). Growing attempt to better link humanitarian and development actors with disaster risk reduction (DRR) efforts and to increase investments in DRR. Overall, coordination seen to improve with introduction of Cluster Approach and positive views about the value of clusters outnumbered negative ones. But leadership was a noted weakness - the Humanitarian Coordinator (HC) system needs strengthening. Other coordination trends highlighted included a growing role for regional bodies (e.g. ASEAN) and agency consortia. Monitoring identified as a continuing weakness. HR improvements (in capacity, quality, and professionalism) were encouraging but Many agencies made real efforts to increase investment in operational capacity and quality of human resources. Improvements in professionalism of humanitarian staff noted, but high staff turnover and a need to invest more in national staff development. There are also growing capacities on the part of national governments to meet the needs of their own citizens in times of disaster in many contexts, which requires greater consideration in advance of launching response efforts.

Connectedness: An unmitigated scarcity of investment in local and national capacities was a repeated theme, as were concerns with the top-down tendencies of the system and the risk of undermining local capacities. However, there are also signs of improvement in how international agencies work with local humanitarian actors, with the survey finding a majority of respondents felt efforts at capacity building had increased in the past two to three years. A clear momentum around need for greater downward accountability and participation, and a growing number of examples of investments in feedback and complaint mechanisms and greater transparency, which benefits programmes.

Efficiency: Efficiency issues, including the risks of corruption, continue to be relatively neglected in literature/evaluations of humanitarian action, although Transparency International is developing an anti-corruption toolkit. -Widespread concern about overhead and programme support costs, particularly in relation to new financial mechanisms. People also noted, however, that constant drive to minimise administrative costs was leading to chronic under-investment in key capacities that could serve to improve performance. Arguably too great a focus on driving down admin costs.

Coherence: focus on: i) whether core humanitarian principles, international humanitarian law (IHL) and refugee law were being respected in humanitarian programming - a real challenge with a noted lack of respect for IHL and the principles in many recent conflicts and integrated or 'whole of government' approaches threatening humanitarian space - both requiring renewed advocacy efforts and more principled action by agencies, and ii) consistency in objectives and actions for protection and for advancing the crosscutting issues of illness, age, gender and disability which are hard to keep sight of once 'mainstreamed'. Improved guidance and awareness but confusion about the concept of protection and who has responsibility. There has been criticism of the quality of protection work, including the deployment of inexperienced staff, breaches of confidentiality of affected populations and inconsistent knowledge and application of relevant laws.

Meyer Foundation: Lessons learned from hiring consultants

The Humanitarian Relief blog pointed out to the following article on the lessons learned when it comes to hiring consultants. Richard Moyers identified three main complaints about the work consultants delivered:

• the consultant didn't understand us
• the consultant was too busy
• the consultant was too directive/not directive enough

Some of Moyer's suggestions for nonprofit organisations is to spend more time verifying whether the consultant is the best fit for the job. Helpful questions in this process include:

• Have you worked with an organization like ours before?
• Have you done a project like this before?
• How many other clients and projects will you be handling while you're working with us?
• How much board and staff time will it take to support your work?
• Do you view yourself as a neutral facilitator, a guide, or a participant in the project?

ALNAP: Key messages from its 8th RHA

The 8th edition of ALNAP's Review of Humanitarian Action series has come out in July 09. As of yet only available electronically, but soon also in printed edition (Aug 09).

The 8th Review contains three in-depth studies.
The first study is on humanitarian performance and provides a wide-ranging overview of the performance agenda - at the heart of ALNAP's work - drawing on experiences from the private, public and development sectors.
The second study focuses on improving humanitarian impact assessment, and provides a comprehensive framework to help bridge theory and practice in operational settings.
The third study is a systematic review of innovations in international humanitarian response, which presents ways to think about and strengthen innovations across the sector.

ALNAP: Humanitarian Performance Project

As part of its Humanitarian Performance Project (HPP), ALNAP is currently piloting a mechanism which aims to provide a regular assessment of system-wide humanitarian performance. The project will compile some fundamental measures of the composition and performance of the international humanitarian system, providing a baseline for assessing future progress. They want to get as diverse a range of actors as possible to participate, especially field-based practitioners, and invite you to complete a short survey which should take no more than 20 minutes. Click here to begin. A French version is also available and can be accessed here.

ALNAP: 8th Review of Humanitarian Action: Performance, Impact and Innovation

The 8th ALNAP Review of Humanitarian Action(RHA)is now available
in electronic form on the ALNAP website.

This edition presents three complementary studies on performance, impact and
innovation.

The first chapter on performance mechanisms is based on the ongoing work of the Humanitarian Performance Project and draws on experiences from the private, public and development sectors in order to locate current data collection initiatives within the current humanitarian performance agenda, as well as providing recommendations on how to improve performance mechanisms in the future.

The second chapter on impact assessments presents 4 case studies from ALNAP member agencies and provides a comprehensive framework to help plan for and implement impact assessments. The framework has already been presented at a number of different fora and used within a number of ALNAP member agencies with very positive reactions.

The third chapter on innovations is the first in-depth study of its type undertaken in the humanitarian system, and provides a way of thinking and promoting positive innovations across the sector. This study will be the substantive theme of the 25th ALNAP Meeting in London on November 17th/18th this year.

Finally the ALNAP website has now been substantially revamped!

Age, Gender and Diversity Mainstreaming Evaluation

From It Begins With Me:

An 8 month evaluation of the strategy to consider persons of diverse age, gender and physical condition in planning and program (Age, Gender, Diversity Mainstreaming or AGDM) for the UN Refugee Agency (UNHCR) begins this week [late March 2009]. In this video independent evaluators Virginia Thomas and Tony Beck join Naoko Obi, head of UNHCR’s Community Development, Gender Equality and Children Section, to discuss the mission, methodology and milestones for this comprehensive task.

ALNAP: Real-time Evaluations of Humanitarian Action - A guide

Real-time evaluation (RTE) is probably one of the most demanding types of evaluation practice, requiring not only a wide range of skills from evaluators but also a tightly focused professional approach in order to meet the time demands of an RTE. However, RTEs are not about doing regular evaluation work faster. Approaches in RTEs must be different from those in regular evaluations because of the limited time available to make an evaluative judgement.

This pilot guide is intended to help both evaluation managers and team leaders in commissioning, overseeing and conducting real-time evaluations of humanitarian operational responses. Drawing on a synthesis of existing good practices, it is intended as a flexible resource that can be adapted to a variety of contexts.

Download the guide from the ALNAP website.

Using Evaluation: Progress and Lessons from the TEC Regional Stream

This publication captures and acknowledges the key contributions made by a wide range of partners in supporting the TEC regional stream effort. The publication demonstrates the value of sharing and learning from system-wide efforts to reorient and improve future DRR, response, and recovery initiatives in and outside the region. Read more on the Reliefweb site.

The paper contains following topics:

• The Need for Humanitarian Reorientation
• Introduction: Spearheading the Regional Stream
• Analysing and Sharing the TEC Process, Findings, and Recommendations
• Dialogue and Evaluation: Two-way Communication towards Reoriented Recovery
• Developing and Disseminating the TEC Message: Influence through Targeted Use of Papers and Publications
• Building Consensus on Fundamental Reorientation in Regional Organisations
• Training: Building Knowledge and Skills for Reoriented Service Delivery
• Expanding Impact through Influencing Policymaking
• A Solution Exchange: Practitioners in Asia Discuss Follow-up Lessons from Tsunami Recovery

BBI: Moving Beyond Rhetoric: Consultation and Participation with Populations Displaced by Conflict or Natural Disasters

UPDATE: Download an additional report, Listening to the Voices of the Displaced: Lessons Learned; author Roberta Cohen reviewed over 800 interviews with IDPs from different regions, stages of displacement and circumstances to analyze the concerns voiced by IDPs. The report is available from Reliefweb.

This desk study seeks to encourage reflection and debate on the benefits, limitations and risks of consultative and participatory approaches in working with communities displaced by both conflict and natural disasters.

It reviews previous experiences of consultation with internally displaced persons and others and explains why consultation is critical for both displaced communities and the agencies which work with them. Advantages of consultation and participation include both instrumental benefits, for example, better needs assessments, improved efficiency, implementation, and sustainability of projects; and value-based benefits, including empowerment and capacity building of affected communities.

Download the study from Reliefweb.

Praxis Paper 21: 'Participatory Monitoring and Evaluation in Practice: Lessons learnt from Central Asia'

Participation in development is generally accepted as a process that is fundamental to addressing issues of ownership and sustainability. Everyone acknowledges the value of participation. Yet when it comes to developing monitoring and evaluation systems many civil society organisations continue to employ experts to develop their objectives, indicators and data collection methodologies – with very little participation from the very people they are attempting to engage with.

This paper records an attempt to develop a fully participative M&E system, drawing on the experience of a team of INTRAC staff working on a civil society strengthening programme in close collaboration with their partners in the five countries of Central Asia.

Praxis Paper 21; 'Participatory Monitoring and Evaluation in Practice: Lessons learnt fromCentral Asia'

ALNAP 7th Review of Humanitarian Action

The 7th edition of the ALNAP Review of Humanitarian Action (RHA) explores critical issues involved in the development of mechanisms for assessing and improving the performance of humanitarian actors. It contains a short introduction and three main substantive studies.

The first study is on organisational change and provides perhaps the first substantive analysis of the experiences of humanitarian organisations in carrying out processes of change and improvement. The second study focuses on joint evaluations of humanitarian action and provides a comprehensive account of current strengths and weaknesses. This chapter also contains a meta-evaluation, which assesses the quality of joint-evaluation reports using the ALNAP Quality Pro Forma. The third study is an evaluation synthesis of key lessons learned from the international response to the 2005 Pakistan earthquake.

Download the review from the ALNAP website. Audio of the launch is available online from the ODI website.

ALNAP & ProVention: Flood Disasters: Learning from Previous Relief and Recovery Operations

This briefing paper provides a synthesis of and introduction to key lessons from evaluations of relief and recovery/humanitarian response programmes to flooding in the last 20 years from Africa, Asia and the Americas. The paper is intended for people working in relief and recovery operations for floods – those who have to decide if, when and how to intervene.

The paper covers lessons for the following key topics which may be relevant in various ways and at different times for flood preparedness, relief and recovery:

Lesson 1 – Flood risk reduction
Lesson 2 – Building ownership and engaging with local capacity
Lesson 3 – Needs assessment
Lesson 4 – Targeting and monitoring
Lesson 5 – Livelihoods recovery
Lesson 6 – Local economy and market
Lesson 7 – Water, sanitation and health
Lesson 8 – Shelter and housing
Lesson 9 – Managing nationwide response and coordination

The paper is available for download from Reliefweb.

Intrac updates: Rethinking Monitoring and Evaluation

INTRAC's latest newsletter ONTRAC deals with 'Rethinking M&E' and included perspectives
on

• the logframe from Oman,
• storytelling M&E from CDRA in South Africa,
• how to combine qualitative and quantitative M&E in QUIP
• Monitoring and Learning and
• Brian Pratt on the increasingly sharp polarisation within civil society M&E.

Click on the link to download the newsletter as a pdf.

The newsletter coincides with the publication of their new book 'Rethinking M&E: Challenges and Prospects in the Changing Global Aid Environment' - we hope to have a copy in the BRC Library soon.

The book 'incorporates the good examples and innovative M&E solutions of 120 development professionals from an enormous range of countries, circumstances and specialisms', '[e]mphasising Southern perspectives and covering a rich variety of experiences, it stresses the important role of M&E in challenging many of our assumptions about poverty alleviation.'

Read more about the book in the longer post:

‘Rethinking M&E’ both analyses practitioner issues and situates them within wider aid trends. It takes as its premise the observation that official development aid is shifting towards an increasingly technocratic, managerial, state-centred approach. It follows that M&E within the aid chain worldwide is directed away from its focus on qualitative outcomes and long-term poverty alleviation impacts. Within this context, ‘Rethinking M&E’ provides innovative insights into such areas as M&E of NGOs as donors, the M&E of advocacy and the M&E of humanitarian emergencies.

Wherever you find yourself in the world of development M&E, this book will present useful experiences from others in similar situations. It shows that there is momentum and energy going into making M&E work for learning, empowerment and poverty eradication.'