Nicole McNeilly
Impact Advisor , Europeana Foundation
For the first time we know of in the cultural heritage sector, in 2020 Europeana piloted the use of Outcome Harvesting in our impact work, an evaluation method increasingly applied in the development sector. Here we share what we learned about the methodology, how we used it and how it could be useful to cultural heritage impact practitioners.
Outcome Harvesting is a methodology developed by evaluator and consultant Ricardo Wilson-Grau and colleagues. It can help you to understand and verify whether change has occurred as a result of your actions in complex and long-term situations where it may be difficult to define specific goals for an activity. This could be, for example, when objectives or goals hadn’t been determined in advance of the activity beginning, or when it evolved over time.
Outcome Harvesting is a six-step evaluation methodology that can be used in more complex situations ‘where relations of cause and effect are not fully understood’. Information about outcomes are ‘harvested’ and analysed across different steps:
Map the users of the Outcome Harvest and identify stakeholders
Define the research question
Map the outcomes (document review, consultation with partners)
Substantiate the outcomes with your stakeholder(s)
Interpret and analyse your results
Use the findings
Unlike simple descriptive outcomes that identify changes in behaviour or attitude (for example, increased likelihood to visit an exhibition), an outcome in Outcome Harvesting is rich and descriptive. It can range anywhere from a sentence to several paragraphs long. It should cover the who, what, why, where and how. The ‘who’ is particularly important - those involved have to be identified to help with the substantiation.
Outcome Harvesting can be used when you are focussed more on outcomes (effectiveness) than activities (efficiency) and when, for example, activities might develop organically over time; when you want to know more about processes of change; and when the relationship between cause and effect is complex. You can find out more on the BetterEvaluation website.
At Europeana, we felt that Outcome Harvesting was a suitable and exciting approach to pilot in our impact assessment of Europeana and EuropeanaTech’s contribution to the implementation of IIIF (a set of open standards for delivering high-quality, attributed digital objects online at scale). The descriptive ‘outcome’ that we investigated was if the adoption of IIIF by the Nationalmuseum, Sweden’s museum of Art and Design (2016-2017) was thanks to Europeana’s promotion of the tool and support from Europeana in implementation.
As this was a new methodology for us we sought advice and coaching on the methodology from an external expert. It took a lot of time and we relied on the participation of partners involved in or connected to EuropeanaTech and IIIF.
We interviewed seven stakeholders in interviews lasting from one hour to 90 minutes. Interviews proved to be an effective way to introduce the interviewees to the methodology, though, interestingly, we learned that Outcome Harvesting most commonly captures written responses using surveys or other methods.
Ultimately, we found that we could not substantiate this outcome in a verification interview. This means that we couldn’t prove that the museum adopted IIIF as a result of Europeana or EuropeanaTech’s actions.
While we couldn’t substantiate our outcome, by asking other questions around the Outcome Harvest data collection, we did, however, have other learnings. For example, we could see in an in-depth case study that digital change can take place very slowly, and that it can take a lot of time to adopt new technology. We identified some of the many different factors that influence an organisation’s digital transformation.
We concluded that without the work of the EuropeanaTech Community, there would be no IIIF data on the Europeana platform. EuropeanaTech increased the momentum and visibility of IIIF amongst the cultural heritage community, facilitated exchange between Europeana, cultural heritage institutions and tech professionals, and provided a space for experimentation and technical advancement. Yet we lack the data that would help us investigate if and how EuropeanaTech contributed to an increase in IIIF-compliant data on Europeana. The lack of data is the result of the time it takes to make new or adapt existing technical infrastructures. This time frame, and the complexity of the context, also makes attributing one factor or trigger for change very difficult.
Our original brief did not fully anticipate the complexity of the research area. Though the Outcome Harvesting methodology was ultimately not suitable for investigating change in this context, it helped us to identify tangible outcomes and check whether these had been achieved with relevant stakeholders. Though Outcome Harvesting is designed for complex situations, we noted that it has limitations when used in a complex context with a slow pace of change and many other influencing factors. Further research, and a longer-term perspective, would be needed to adequately address the research question we asked.
You can read more detail about how we applied Outcome Harvesting in the EuropeanaTech impact assessment. We’ll also be publishing this ‘how-to’ guide in the next version of Phase two of the Impact Playbook later this year. Join the Impact Community to be the first to hear about it!
