V4Design

From 2017 to 2021, the V4Design project has worked to lower barriers to access to digital cultural heritage objects for architects, creative designers and video game creators. These communities are often overlooked by the cultural heritage sector, but are a powerful creative force when reusing and remixing cultural heritage to power bold new games, architectural designs, or other creative projects.

The project focused on fostering re-use of digital cultural heritage by collecting and providing more than 370,000 objects from Europeana, and thousands more objects from providing partners and public web resources, to the V4Design database. The project re-purposed content by developing novel approaches that allow for: 3D reconstruction and modelling of objects from video content; Machine Learning-powered localisation of buildings and objects in images and video content; extraction of aesthetic patterns and architectural and artistic styles from images and videos; enrichment of 3D object metadata with semantics and explanatory text descriptions. The project deployed innovative architecture, design and Virtual Reality (VR) game authoring applications.

The work done throughout the V4Design project has allowed creative designers to reuse Europeana content in myriad ways, serving the need for cultural heritage as input into new creative projects. In this way Europeana aimed to reach creative communities that are otherwise often overlooked by Galleries, Libraries, Archives and Museums.

Explore the final showcase of the results.

Infrastructure innovations

V4Design focused on a few key innovations to support the reuse of the 3D cultural heritage content. 

1. V4Design Crawler: The project developed a tool that collects digital cultural heritage content from relevant web sources and social media platforms in a single module. It takes different inputs and outputs them as a single interoperable metadata standard: SIMMO. It applies various techniques (including scraping, retrieval from API) to integrate the most popular websites and platforms that share openly licensed cultural heritage.

2. Aesthetics Extraction: Project partners used machine learning techniques to extract aesthetic concepts out of architecture and paintings, and train  classifiers to recognise certain aesthetic styles in images. 

3. Style transfer: V4Design developed a technique that allows people to recreate the content of an image by adopting the extracted style of an image depicting a painting, advancing this technology over the timeframe of the project.

4. Spatio-temporal building and object localisation: Using Computer vision techniques, V4Design developed machine learning models that can detect and annotate several parts of images and videos. The models can easily distinguish the facade of a building from its background and surroundings, and then tag the different parts of this building with textual metadata tags. A separate scene recognition module, characterises the images or video frames as indoor or outdoor scenes. If the scene is outdoors, then the surroundings of the building are automatically analysed for more information. If the scene is indoors, another model tries to locate and annotate objects that might be indoors: furniture, decoration, etc. These models all work together to assist the 3D reconstruction of these identified objects by removing unwanted clutter and increasing dense reconstruction performance. For this purpose, deep learning models were trained and extended, using specialised datasets fully aligned with the V4Design’s scope.

5. 3D Model Reconstruction: The project developed an automated video processing pipeline for pre-existing video footage which makes consecutive frames suitable for reconstruction and separates video from multiple angles. The project also  used an alternative method for handling medium to large sized image collection datasets, involving a pre-trained feature vocabulary to detect similar images in a much faster manner. By using the inputs of the style transfer module, the project could restyle the texture of the models.

6. V4Design authoring tools: Through the project, people using  Unity or Rhino3D, two widely used and popular 3D editing applications, now have access to a plugin that makes re-using cultural heritage as easy as pressing a button. Using either plugin, one can search through the thousands of available 3D models and import them into their project with a click.

Explore all of the innovations in the V4Design project on the project website, where you can find a rundown of all the innovation that took place throughout the project in a useful factsheet.

V4Design was a Horizon2020 project funded as a Research and Innovation Action.

Project partners 

• The Centre for Research and Technology-Hellas (CERTH) - Information Technologies Institute (ITI) - Project Coordinator

• KU Leuven

• Universitat Pompeu Fabra (UPF)

• Robert McNeel & Associates

• Herzog & de Meuron (HdM)

• The Aristotle University of Thessaloniki (A.U.Th.)

• Solaris Filmproduktion GbR (SFd)

• ArtFilms Ltd (AF)

• Deutsche Welle (DW)

• Europeana Foundation (EF)

• Nurogames (NURO)

Find out more about the V4Design Consortium.