Some things we care about for NFDI4Culture – Or: We want better tools and workflows for annotating images and 3D models!

by Ina Blümel, Lucia Sohmen and Lambert Heller 

The NFDI (national research data infastructure project, German: Nationale Forschungsdateninfrastruktur) is a funding programm by the GWK (Joint Science Conference, German: Gemeinsame Wissenschaftskonferenz), which aims to improve quality and accessability of research data. So far, the TIB has been part of three successfull proposals by consortias in different research areas. The Open Science Lab represents the TIB in NFDI4Culture, the consortium for architecture, art history and media studies.

In the context of our work packages in the TIB Open Science Lab, we are working on the bundling and updating of current developments in describing digital cultural data, for example images and image content: annotations.

Are we really only interested in images? No, in the long run it can and should also be about other types of objects – in the context of 4Culture especially about music notations, audio recordings, films, dance and theater notations, and 3D models. (The latter is also important to us because we represent architecture in 4Culture).

There is much to be said for making approaches and tools that already exist in the field of image and 3D annotation available to the various professional communities of 4Culture as early as possible, and to develop them further together with them. We want to make sure that much of what is created in the process will very soon be used to address at least the other types of objects mentioned above.

1. Beyond the neutral (encyclopedic, so to speak) description of images and the general definition of image content, we believe that image annotations should be made easily communicable in different contexts by all those interested in images – from the informal description for one’s own peer group or the description in the protected space of a course (scenarios that are supported in an exemplary manner by Hypothes.is, for example), up to the citable micro-publications – the latter either decentralized and merely following web standards, or made identifiable by centrally registered identifiers such as ORCID (for the author of the micro-publication) or DOI (for the micro-publication itself).
2. We believe that freely available neural networks from the field of image recognition should be available as a toolbox in the image annotation workflow – for example, to alert annotators to similarities between sections of different images, or to recommend specialized vocabulary for describing image content. In this context, we also explore the impact of AI-supported proposal tools on the work of cultural data enthusiasts.
3. Expansion of the approach from the ISA tool to include objects that are not (yet) found in Wikimedia Commons. The Wikimedia community is wonderfully supporting people interested in images with a simple user interface to find suitable entities from the open, controlled vocabulary of Wikidata, and to build semantically structured, weighted statements from these entities, especially about image objects, and to create simple, multilingual image descriptions. We have already developed tools for image annotation in one of our 4Culture predecessor projects.
4. Open, controlled vocabularies such as Wikidata could in future be supplemented by more specialized, subject-specific vocabularies such as Iconclass – based on Wikibase, as part of a networked community of subject-specific Wikibase instances inside and outside NFDI. We see ourselves confirmed and inspired by the enthusiasm of the NFDI community about Wikibase. We plan to blog more about this very soon. We have already had good experience in projects like GESAH in providing professional communities with a range of suitable vocabularies for describing image objects. With this approach we want to use the full potential of the Open Research Knowledge Graph for cultural knowledge as well.
5. We believe that standards such as IIIF help to link annotations precisely to image content or image details, which could help to support new ways of curating and rendering selected image content. In the aforementioned GESAH project we are already gaining experience with IIIF viewers for digitized images.
6. We assume that open standards for content identification like the ISCC especially will help to make annotations portable. An annotation made on a photograph of a painting can already be matched with another photograph of the painting with high certainty. This allows, for example, to automatically retrieve, show and reuse it in the context of Wikipedia or a museum website – provided that the annotation itself is freely licensed, which should always be suggested in the annotation workflow anyway. In addition, users should ideally be able to perform annotations in their familiar environment, wherever it is located, while still making their annotations easily findable on the web and reusable by third parties.
7. We noticed that especially for 3D models, the step-by-step description and completion of models, and the visualization of such enrichments, is an application scenario that has been studied for some time, but is still far from being as widespread and accessible as would be desirable for collaborative creative processes. The use of machine learning can be helpful here, for example to visually combine photos and sketches with existing models.

Ina Blümel

Stellvertretende Leitung Open Science Lab und HsH-Professur für Vernetzte Daten in der Informationswissenschaft