How the Horizon Report Museum Edition was developed

The Horizon Report 2012 Museum Edition, which mentions Europeana as an example of ’Open Content in Practice’, is an influential publication that focuses on technology use in museums and is read by over one million people a year. The work that put it together took place on a wiki and could be described as 'crowd-sourced research'. In this extended blog post, Dr Susan Hazan, Curator of New Media at the Israel Museum, Jerusalem, describes the reality TV-style processes and the brave choices the Board faced. Dr Hazan is on the Horizon Project’s Advisory Board and is also is a Europeana Network member, a member of the Europeana End-User Research Workgroup and is Co-Chair of the User-Generated Content Task Force.

Image from the cover of the Horizon report. © Harold M. Miller, courtesy of The Children’s Museum of Indianapolis

The Horizon Project Advisory Board came together last September with project leader Dr. Larry Johnson (Chief Executive Officer of the New Media Consortium, Horizon Project Founder and Co-Principal Investigator), Koven Smith (Project Co-PI) and Dr. Holly Witchey (Editor and Co-PI), to produce the second Museum Edition of the Horizon Report. The group of multi-disciplinary thinkers, including 42 curators, educators, web developers, and cultural heritage specialists, focused on the application of emerging technologies for museums. The Report dealt with digitally mediated platforms and experiences drawn from museum practice around the world with the goal of surveying them for their educational and interpretation qualities.

Described as the consensus sapientum (unanimity of the wise), the Horizon Project is a qualitative research project that crowd-sources the expertise of the group in order to identify museum education technologies, to rank their potential impact for digitally mediated interpretation, and to map them across specific timelines. The goal of this process was not to prescribe tools, but rather to highlight those technologies with the greatest potential for education and interpretation. As the intense research progressed, the group whittled down the list to eventually reveal the six key emerging technologies that would influence museum education over the next five years.

The result of this process was a report on the projects that promised the widest impact and potential for adoption from across the museum sector.

Time-to-Adoption Horizon: One Year or Less

Mobile Apps

Social Media

Time-to-Adoption Horizon: Two to Three Years

Augmented Reality

Open Content

Time-to-Adoption Horizon: Four to Five Years

The Internet of Things

Natural User Interfaces

The previous Museum Edition had been downloaded more than 25,000 times and the Board expected an increased readership the following year. With such an impressive track record of readership, this suggested a significant impact on strategic technology-planning in museums in dozens of countries around the world. NMC estimates that the readership of the Horizon Report - covering various aspects of education - stands at over a million people a year across all its various editions and spans more than 150 countries.

The work took place via a wiki. As we logged in, we discovered extensive resources alluding to the work up ahead: a hoard of news and press clippings; Delicious.com-tagged resources; links to our previous editions of the report; descriptions of dozens of emerging technologies, and much, much more! Those of us who had participated in the previous Museum Edition helped the others to understand that we were growing this resource together as we worked. For the newcomers, as I remembered from the previous year, this must have looked like quite a daunting challenge.

We started out by reviewing the articles, reports, press clippings and other materials on the site, adding our own as we ploughed through the maze of information and ideas. The material had already been carefully culled by the project staff from hundreds of leading sources. The background reading was both challenging and inspiring. We tagged the press clippings with our signature by simply adding 4 tildes to the end of the article, like this ~~~~. This made the popular content more visible because as soon as the signatures started to populate the articles, it became clear where the consensus was emerging.

There were, in fact, a couple of hundred links in our secondary research, all annotated and organised into seven categories:

Emerging Technologies
Challenges and Trends
Published 'Technologies to Watch' Lists
Technology in Popular Culture
Reports and Research
Miscellanea - Stories, Examples, Food for Thought
Essays and Interviews on the Future

We were asked to review the headlines in each category and to select 2-3 items that we thought our colleagues on the Advisory Board should definitely read. As we continued with our evaluation into our chosen sections, we again added our 4 tildes (~~~~) to indicate what we thought was worth sharing. Although this was fairly labour intensive, the material was so fascinating that it was difficult to ‘switch off’ and go back to our usual work duties and day jobs.

According to the editorial team, the work was meant to accomplish two things from a research perspective:

To provide a quick and dirty way to reduce the set of background readings for the group with the sorted list persisting as an artifact of the work that can be shared with others.
To improve the reliability of the responses the group would put to the research questions once the Advisory Board had crowd-sourced the data.

Once again, we were invited to add our own clippings, comments or examples of our own work to further enhance the data set before answering the research questions on the wiki. At this stage, we were asked to select around 5 topics in order to identify the critical challenges museums face when making technology choices. The next stage was designed to capture key trends that we considered were fuelling the development of technologies of interest to museums. At all stages we were encouraged to add our own comments or URLs to the list if we felt that anything was missing.

The final and most critical stage was the ranking of the research questions.

Dealing with each of the research questions in turn, we were asked to identify technologies that we judged will be important along each of 3 adoption horizons. Our votes were divided into 3 sub-questions, one for each adoption horizon and we were allowed to assign multiple votes per horizon according to our own assessments. The way we used our votes was up to us – we could allot all 10 votes to a single item, 2 votes for one or more items, or spread out all of the 10 votes equally. As we whittled down the list, voting for a specific topic on one horizon, it was no longer available on any of the other two horizons.

Then we moved on to the short list. Using the 'Horizon Project Voting Tool' we tackled the 12 semifinalists for the report, along with 10 key challenges and 11 key trends. Our goal was to eliminate just one item from each of the horizons leaving three still standing in each horizon, and one each from the trends and challenges in a reverse-ranking process.

Although we had spent many hours reading and mulling over the material up to this point, the voting process was reminiscent of a reality show and really quite fun. We were cued to start the voting process like this…

'Welcome to SURVIVOR! Horizon.Museum Edition. The tiki lamps are lit, and the Advisory Board is assembled in virtual space. Each of you now faces a set of difficult choices.

(cue tom-toms)

Which of the items listed within each of our three adoption horizons do you think is weakest and *least* deserving of inclusion in the report? And which of the trends and challenges?

You will decide...

(cut to screen shots of the web)

(louder tom-toms, camera close up, pans Advisory Board members' faces as the narrator intones...)

...which of these are not going to survive?'

(cue discordant musical flourish; quickly fade to black)

Clicking our way through the topics, challenges and key trends we bravely made our choices; chopping away at the ones we felt least relevant and laying bare the ones who were to survive. I found the process fascinating and now we hope that you will find the Report interesting and useful.

More information

The Report is available on: http://www.nmc.org/publications/2012-horizon-report-museum

The NMC Horizon Report: 2012 Museum Edition is a publication of the New Media Consortium and the Marcus Institute for Digital Education in the Arts

Citation
Johnson, L., Adams Becker, S., Witchey, H., Cummins, M., Estrada V., Freeman, A., and Ludgate, H., (2012). The NMC Horizon Report: 2012 Museum Edition. Austin, Texas: The New Media Consortium.