by Scott Bayley
Evaluation for Accountability or Learning?
A comparison of two primary evaluation purposes
Published on Evaluation Capacity Development Group, 17 July, 2017. http://www.ecdg.net/evaluation-for-accountability-or-learning/
Evaluation can be defined as the systematic empirical examination of a program’s design, implementation and impacts with a view to judging a program’s merit, worth and significance. Evaluation reduces uncertainty for stakeholders, aids decision making, and serves a political function. Evaluations are commonly undertaken for the following reasons: policy development; improving the administration/management of a program; demonstrating accountability; and facilitating stakeholder participation in decision making.
Evaluation studies are intended to add value for program stakeholders. If the role of the private sector is to generate profits in the context of changing market forces, the role of the public sector is to create value in the context of changing political forces. Guiding questions for an evaluation department within an aid agency include:
- Who are our clients?
- What do they want/need?
- How can we create value for them?
- How will we monitor our results?
- How has our program responded to the lessons being learned?
Ever since the evaluation of aid programs first began in the 1960s there has been tension and controversy over using evaluation for accountability purposes versus emphasising evaluation to support organisational learning. Donors tend to value the accountability function while program staff are more interested in learning how to improve operational practices. The way in which these two goals are approached has a major influence on how evaluations are organized and conducted. The underlying problem is that it is not really possible to fulfill both purposes in a single evaluation study. The two approaches are often incompatible as shown in the following table:
| If accountability is the goal: | If learning is the goal: | |
| Selection of individual evaluation topics: | Based on size of program budget, perceptions of performance problems | Based on the potential for learning helpful lessons |
| Primary audience for the evaluation: | Donors | Operational staff |
| Objectives of the evaluation: | Accountability, enhanced control, informing resource allocation decisions, a summative focus | Program improvement, enhanced understanding, a formative focus |
| Focus of the evaluation: | Issues relating to outcomes and operational compliance | Varies according to the program’s stage of development and the information needs of identified stakeholders |
| Basic orientation: | Retrospective | Prospective |
| Timing of the evaluation: | Conducted late in the program’s life cycle | Conducted early/mid point in the program’s life cycle |
| Relevant evaluation models: | Objectives based, decision based, needs based, performance auditing, cost benefit studies, autocratic models | Responsive, utilisation focused, case studies, constructivist, democratic models |
| Stakeholder involvement: | Generally limited to commenting on the terms of reference, serving as a source of data, and responding to recommendations | Active involvement in all stages of the evaluation, stakeholders are often organised into evaluation committees (steering, advisory, consultative, data interpretation) |
| Relationships between the evaluation team and program staff: | More adversarial | More collaborative |
| If accountability is the goal: | If learning is the goal: | |
| Evaluation methodology: | Preference for quantitative methods | Multiple methods |
| Approach to sampling: | Representative - probability based sampling | Purposeful - sampling information rich cases |
| Report content: | A focus on identifying problems and shortfalls (the gap between expectations and actual achievements) | A focus on what is currently working well, what can be learned from current practices, and how further improvements can be made |
| Reporting style: | Technical style like a journal article, a heavy emphasis on text | Audience friendly reports in plain English with lots of diagrams, multiple reporting formats, an emphasis on face to face dialogue with intended users |
| Recommendations: | Typically based on reverse problem logic. If ‘X’ is found to be broken, the recommendation is to ‘fix X’, limited consultation with program staff | Based on the logic of program planning. Additional data is gathered concerning the technical, financial, legal, political, and administrative viability of potential program improvements with the active collaboration of intended users |
| Promoting utilisation of evaluation findings: | Limited engagement and ongoing support provided to potential users | Active engagement and ongoing support provided to intended users. A focus on supporting learning processes. |
| Resources devoted to disseminating evaluation results: | Typically less than 5% of the evaluation’s budget | 25% of the evaluation’s budget |
| Values of the evaluation unit: | Emphasis on being objective and independent / impartial | Emphasis on generating relevant contextually based knowledge and adding value for identified stakeholders |
| Evaluator is perceived as: | A policeman, auditor | A consultant, teacher |
The relative emphasis given to accountability versus learning in an individual study and in an evaluation office’s overall work program is vitally important because this determines the way in which evaluations will be conducted and which stakeholders will be served (and which stakeholders will not be served).
Once again:
- Who are our clients?
- What do they want/need?
- How can we create value for them?
- How will we monitor our results?
- How has our program responded to the lessons being learned?
Scott Bayley
Principal Specialist, Performance Management & Results
Department of Foreign Affairs and Trade (DFAT)
Canberra, Australia
17 July 2017
Affiliations
Fellow of the:
