FO Manual Annex B: Results framework indicators

Progress against the programme results framework will be measured with the use of indicators.  Indicators are a means to quantifiably measure progress and indicate the extent to which an outcome or output has been achieved.  

Most indicators are quantitative, in that they measure the quantity of things like the number of goods/services delivered, number of people supported, or number of buildings improved.  

Some indicators are qualitative in that they measure people’s opinions, perceptions, and attitudes. Nevertheless, qualitative indicators can also be expressed in quantitative (numerical) terms, for instance by using scales. 

In most instances, a set of indicators is required to measure an output or an outcome satisfactorily. This is because each programme has multiple projects, and each project has multiple activities, all of which contribute to programme-level results in various ways. It is recommended that each outcome and output has at least two indicators.  

Typically, multiple projects will contribute to each individual indicator. However, not all project-level results are captured in the programme results framework because this would result in too many indicators and create disproportionate data collection and reporting burdens. Specific project-level results should be reported in the Project Level Information section in GrACE. 

The potential difficulty of collecting data and reporting data should be kept in mind when selecting indicators. Often the issue is not how many indicators there are in the results framework, but how difficult it is to report on certain indicators. 

With the support of the FMO, the Fund Operator should use their judgement and experience when selecting the most important indicators to include. The suggested maximum number of indicators for most programmes is 30 indicators (not including the bilateral indicators). Programmes can have fewer indicators.

Core indicators

In the Financial Mechanisms 2014-2021, core indicators proved to be a valuable library of metrics. When multiple programmes use the same indicators, it allows the FMO to aggregate results, which helps to tell the results story of the entire Civil Society Fund.

The FMO will create a library of Civil Society Fund core indicators library for the 2021-2028 period. The library will be developed based on indicators that have worked well in the previous period and has been aligned with relevant indicators from the EU and Sustainable Development Goals. Each suggested indicator is mapped against an outcome or output of the Civil Society Fund results framework, for ease of use.   

The Fund Operator is encouraged to use the core indicators library as a ‘menu’, selecting the most appropriate/relevant indicators for their programme, depending on how they have contextualised the Civil Society Fund results framework to their country. The FMO does not set the number of core indicators to include in a programme results framework, but these indicators should be used wherever relevant to help the Fund Operator effectively monitor the progress and track the achievements of their programme. This also helps bring the Civil Society Fund programmes in each country together so that we can see how the programmes are contributing to a vibrant and resilient civil society across countries. 

The Civil Society Fund core indicators library will provide a brief description and methodology for each core indicator, ensuring a consistent approach to data collection and allowing results to be aggregated across the Grants.

Custom indicators

When there is not a suitable indicator in the core indicators library for measuring a particular outcome or output, the Fund Operator can develop custom indicators with the support of the FMO. ‘SMART’ is a helpful acronym for designing good indicators. Using SMART principles when defining indicators ensures they can effectively measure outcome and output level results.

Specific:         Indicators should be specific and clearly defined, with a clear meaning and scope. The indicator
should be focused on a specific aspect of the programme or project, rather than being too broad or vague.

Measurable:  Indicators should be quantifiable and measurable with a clear unit of measurement in the indicator statement. Data should be measured consistently by different projects over time, helping to ensure that it is reliable. Data collection should not be too costly, complicated or time consuming to collect and report. 

Achievable:   Indicators should be feasible and achievable. This also relates to setting reasonable indicator targets.

Relevant:       Indicators should be relevant to the output or outcome that it falls under. It should help indicate the extent to which a specific outcome or output has been achieved.

Time-bound:  Indicators should be time-bound, with a clear timeframe for measurement. Typically, results should be reported annually, with final results reported by the end of the programme. 

An indicator should be expressed in neutral terms, not indicating the direction of change (increase or decrease), nor embedding a target. For example, ‘An increase of 30% in the percentage of domestic violence cases prosecuted’ should be reformulated to ‘Percent of domestic violence cases prosecuted’. The direction of the change will be clear when comparing the baseline with the target.

Output indicators tend to measure things such as the quantity and quality of goods or services directly delivered. Therefore, output indicators often have a unit of measurement that is ‘cumulative number’ or ‘binary’. For instance, an output indicator could be ‘Number of CSOs participating in consultations with a public decision-making body on the revision of the children’s rights law’. Results are typically reported for output indicators earlier than outcome indicators because outputs contribute to the achievement of outcomes. 

Outcome indicators measure the higher-level changes that we are most interested in, particularly the effect on target groups. Outcome indicators often take longer than output indicators to report results. This is due to the time lag from when the output was delivered/produced and the changes it contributed to producing. Outcomes tend to be reported when projects are nearing completion. For instance, an outcome indicator could be ‘Number of new/revised national policies or laws in line with international/European human rights standards influenced by CSO advocacy’.

Where possible, try to have a logical link between output and outcome indicators. For example, if a programme implements awareness-raising campaigns, there may be an output indicator ‘Cumulative number of people reached by awareness-raising campaigns’. A more important outcome level result is the actual change that the awareness campaigns triggered. For instance, the campaigns may aim to increase people’s knowledge, change their attitudes, or positively influence people’s behaviour. Therefore, a related outcome indicator should be included to capture the actual changes that the awareness-raising campaigns aimed to make. This could be ‘Cumulative number of people with changed/improved knowledge from the supported awareness campaigns’. 

Indicators for bilateral relations

The programme should contribute to strengthening bilateral relations between CSOs in the Fund Operator’s country and CSOs and other entities in the Donor States, in line with the overall objectives of the Grants. Indicators for bilateral relations measure the bilateral results of all projects in a programme that involved such partnership. All programmes will use the same indicators for bilateral relations, as described in the forthcoming core indicators library. The bilateral indicators are based on the widely used criteria of relevance, efficiency, effectiveness, impact, and sustainability. Additionally, indicators have been included for the level of involvement, trust and overall satisfaction. Together, these indicators systematically capture the broad dimensions of bilateral relations at project level. 

The data source will be a short online survey administered by the FMO to Project Promoters and CSOs and other entities in the Donor States at the end of each project (ex-post). Additionally, these entities will also be asked about the activities that were done in the project partnership. Projects that do not have a bilateral relations component will not receive the survey. Baselines are not applicable (N/A) for these indicators because partners typically do not have sufficient experience of each other at the outset of a project. All programmes will have the same bilateral indicator target of 4 out of 5. When reporting results, indicators will be disaggregated by Beneficiary State/Donor State so any differences may be analysed. 

Indicators for regional civil society initiatives

Regional civil society initiatives are activities that contribute to the objective of the Civil Society Fund and aim to promote regional exchange and networking, with a view to strengthening the civil society sector across the Beneficiary States, sharing knowledge, promoting mutual learning, adoption and use of knowledge and good practice across civil society. The core indicators library will include suggested indicators for regional civil society initiatives, which the Fund Operator may include in their programme results framework should such initiatives be included in the programme.   

Disaggregations

In some cases, there may be a wish to report on how a programme or project has affected more than one target group, for instance women, children, Roma or LGBTIQ+ individuals. There are two possible approaches to measuring results for different target groups: creating two different indicators or adding a disaggregation for a given indicator. 

In this Financial Mechanism, we recommend using disaggregations sparingly because experience has shown that it is difficult to consistently collect and report disaggregated data and that disaggregated data is often underutilised. ‘Gender’ is one type of disaggregation that may be appropriate for certain indicators. To ensure disaggregated data is collected by Project Promoters, it is suggested that this requirement is stated in the text of Calls for Proposals.

Whenever a programme or project targets a specific group, such as women, children, Roma or LGBTIQ+ individuals, we recommend defining a specific indicator to measure a change in this group, rather than using a disaggregation. This should ensure that it is given sufficient attention in data collection and reporting. The FMO will support the Fund Operator to determine when it may be appropriate to include an indicator disaggregation. 

Unit of measurement

To avoid confusion and ensure data consistency, the programme results framework should state the unit of measurement for each indicator. There are several types of units of measurement (see the table below). The most common type is ‘Cumulative number’, which is often the easiest to report on. The unit of measurement should be apparent from the indicator label itself. For instance, ‘Number of…’, ‘Percentage of’… ‘Level of…’ Etc.

Units of measurement for indicators

Data sources

Each indicator should have a ‘data source’, which is to be used for reporting results. Common data sources are ‘Project Promoter records’, ‘Survey results’, ‘Attendance sheets’, ‘Copies of contracts’, and ‘Audio/video/print material produced’. Many other data sources are possible. The important thing is that there is evidence that a result has been achieved. 

Baseline 

In the programme results framework, there is a column for the baseline value for each indicator. The baseline is the situation before a programme or project begins. The baseline is essential for measuring progress and achievements. Without knowing the starting point, we cannot measure the result.

Most output indicators will have a baseline of zero, particularly when their unit of measurement is ‘cumulative number’. This is the case when goods or services (e.g., an advocacy campaign) are yet to be produced, and people are yet to be supported or mobilised.

Outcome indicators also often have a baseline of zero, although they are more likely to have a non-zero baseline than output indicators. The Fund Operator should carefully consider the availability of data for setting non-zero baselines. In some cases, reliable secondary data from governmental statistical agencies, universities, NGOs, or businesses can be used for setting a baseline.

However, if there is no secondary data available for a baseline, or if available data is from a different sample with different characteristics (national vs. local, for example), the Fund Operator may need to conduct (or externally commission) a baseline study. Baseline values may be collected by an agreed deadline specified in the Programme Implementation Agreement.

It is important that the method used for collecting baseline data will be the same as method for collecting results data in future. Results need to be compared with the baseline in a consistent way to measure a valid change. 

Targets 

Indicator targets are values that help indicate whether the programmes outputs and outcomes have been achieved by the end of the programme period. The indicator targets help orientate the programme to the achievement of the outcome and output statements. It should be remembered that the most important results are the achievement of the output and outcome themselves, whereby the indicators indicate the extent to which (some of the) results have been achieved. 

Setting indicator targets before projects are known is one of the main challenges in Results-Based Management. There is often a tension between setting easily achievable targets and setting ambitious targets that stakeholders will strive to meet. After all, when motivated and focussed on ambitious targets, programmes can often achieve more than they imagine. 

The process for how target values are set is outlined in the figure below. In the Programme Implementation Agreement (PIA), targets are initially set as ‘To Be Determined’ (TBD). This is because projects have not yet been contracted. Thus, the specific expected results, expected reach and size of the project, and their selected indicators and targets, as well as the number of projects contributing to programme level targets are unknown. The Fund Operator will ask project applicants to specify target values for the selected indicators in their application form. Once the project has been contracted, the Fund Operator will register the project by filling in the Project-Level-Information (PLI) in GrACE. In the PLI, the Fund Operator will indicate whether the project contributes to the programme indicator targets (a ‘contributing project’) and, if so, will specify the target values for the selected indicators. GrACE will then automatically aggregate the targets of ‘contributing projects’ to identify the programme level target. This will be done periodically as projects may be contracted at different times during the implementation period. The aggregated value of targets will replace ‘TBD’ in the programme results framework. The programme results framework will therefore get automatically updated and finalised. A notification will be sent to the Fund Operator. No modification of the Programme Implementation Agreement will be required. 

Process for target setting in the programme results framework

It is normal for programmes to over-achieve results for some indicator targets and under-achieve for other targets. Most programmes will not achieve every indicator target and there is no expectation that they must do so. In fact, if a programme easily achieves or exceeds all its targets, this may mean that the targets were not very ambitious. A programme may encounter many challenges that will affect programme implementation and results. For instance, during the 2014-2021 Financial Mechanisms, several unlikely risks materialised and affected results, including the Covid-19 pandemic which affected the movement of people and supply chains, the war in Ukraine which affected energy prices, and a historic spike in inflation. 

When targets are unlikely to be met, the Fund Operator should simply explain the reasons in their reporting, so stakeholders can learn lessons for the future. In some cases, it may simply be that the original assumptions when setting targets were incorrect. In other cases, it may reflect the non-performance of certain projects a Fund Operator only has indirect control over. 

The FMO follows a ‘best effort’ approach to achieving targets. This means that the FMO will not impose any penalties for not achieving targets, provided the Fund Operator can show that the programme was implemented according to the Programme Implementation Agreement, and there has been no breach of the principles of implementation and values of the Grant.

Example of filled results framework in Programme Implementation Agreement

*Note: The above example is a partial results framework only and shows only a few rows for illustrative purposes.

The Civil Society Fund programme result framework in table format