Comment on page
Developments and innovations to Goobi workflow in June 2021
- Outsourcing of some statistics to plugins with refreshing of the user interface
- Implementation of a new search syntax
- Revision of the internal folder structure to comply with the Maven convention
- Expansion of the plugin documentation
Goobi now supports the conversion of the metadata format LIDO to METS. For example, it is now possible to import a LIDO file and save it internally as a METS file. In the same way, an existing METS file can also be exported again as a LIDO data set.
Conversion from METS to LIDO and back
Previously, images in LIDO were expected as
With this month's developments, it has now become possible for images to be specified as
<resourceID>instead. This is particularly helpful for data transfer from the MuseumPlus application:
Please note: If an image belongs to a sub-element, the specification must be made within the
<lido>element that describes this sub-element.
Recently, we had extended the so-called variable-replacer to the effect that it now also allows IIIF URLs for the corresponding images of a process in script calls. We had to make some corrections here so that the encoding of the URLs fits the respective use cases.
The vocabulary management in Goobi workflow makes it possible to manage controlled vocabularies in the user interface and to use them as choices for metadata fields in the metadata editor. A particularly practical feature is that the description for a particular entity can be changed and, by linking to only one vocabulary entry in the METS file, the entry is also automatically updated in all processes. Because of these advantages, vocabulary management is being used in more and more projects.
Goobi's vocabulary management is finding more and more use
A problem can arise when using the vocabularies if, after exporting the data, Goobi viewer also needs to access the same vocabularies, as the Goobi workflow installation is often inaccessible from outside the facility network because it is located behind a firewall. For this reason, we have developed a new vocabulary server to which Goobi workflow mirrors all changes to the vocabularies configured in this way. In such a case, the URL of the data records in the publicly accessible vocabulary server, which is hosted by intranda and accordingly accessible worldwide, is entered in the metadata of the METS file. This means that the Goobi viewer can then access the vocabularies without any problems and always receive up-to-date data records from the vocabulary.
The code for this server has not yet been published and can be found in the intranda-internal Git repository: https://gitea.intranda.com/intranda/goobi-authority-server. The still internal documentation can also be found there in the repository: https://gitea.intranda.com/intranda/goobi-authority-server/src/branch/master/goobi-authority-server/docs/AuthorityServer.md. We have not yet decided when to publish this project. Before that, we would like to review the source code and also prepare and translate the documentation.
Since the first days of Goobi, there has been the possibility to create processes based on a catalogue query and thus reuse the data from the catalogue. Unfortunately, however, the response from the respective catalogues is sometimes not such that Goobi can reliably interpret, for example, the publication types from the expected fields. For this reason, it has long been possible to manipulate the catalogue response with the so-called Beautifier before evaluating it. This was already possible in the past with PICA records. What is new is that these possibilities now also exist for MARC records. Such a configuration could look like this, for example:
In this example, the leader at position
19is replaced with a space (
\u0020) if the value
eis in the leader at position
<setvalue tag="leader19" subtag="" value="\u0020">
<condition tag="leader6" subtag="" value="e" />
This example shows how to create the field
<controlfield tag="999">with the value
KXPfor a record of an OPAC query.
<setvalue tag="999" subtag="" value="KXP" />
This example shows how a field content can be changed. Here the field
041$ais replaced with the value
gerif it contains the value
<setvalue tag="041" subtag="a" value="ger">
<condition tag="041" subtag="a" value="lat" />
The following URL documents the configuration for catalogues and also the definition of the Beautifier:
And the wild ride continued this month as far as accessibility optimisations are concerned. One gets the impression that this is a life task. But at least it feels like we are on the home stretch. Let's see how it looks next month. :)
Accessibility is more important than many think!
https://github.com/intranda/goobi-workflow/commit/bcc76fa67841af7a7fe2a50a1fe3398c12554d1c https://github.com/intranda/goobi-workflow/commit/70f65c14ab97a2394a7d5b0d7baa98c6bc433af2 https://github.com/intranda/goobi-workflow/commit/b6bd4dad318910aa8d0aa98a8e8ea0d54ba469a3 https://github.com/intranda/goobi-workflow/commit/17b88937cc79b71c1914533aa85b3685c5ca01fc https://github.com/intranda/goobi-workflow/commit/5806a5e8e5e011d2ac5ac6c68da2dfa950ec4a01 https://github.com/intranda/goobi-workflow/commit/106283895a5604de04c809d2e3a7bcfa99d8fc93
Compiling Goobi workflow is no longer limited to the already quite old version 8 of Java. Instead, from now on not only operation with Java 11 is possible but also compiling.
Excerpt from the console after compiling Goobi workflow with Java 11
For those who would like to compile Goobi themselves, here is a mini-summary of how to do it yourself:
# install needed packages
sudo apt install git
sudo apt install maven
# checkout Goobi workflow from GitHub
git clone https://github.com/intranda/goobi-workflow.git
# go into the Goobi workflow project
# compile Goobi workflow
JSON Web Tokens (JWTs) are being used more and more in Goobi workflow to allow other programmes or people to perform certain actions for a certain time. Until now, a
JWT-Secrethad to be pre-configured manually in order to be able to use this functionality. As of this month, Goobi workflow generates the JWT secret itself if it has not yet been pre-configured.
In the standard installation, the JWT secret is still commented out and can be configured individually.
During the implementation of the automatic generation of the JWT secrets described above, a potential security gap was discovered: The JWT keys were not rotated, which could have allowed an attacker to guess the secret if many messages were intercepted. This has now been changed and the keys are rotated every 24 hours.
It happens very rarely that file names of images of individual processes are renamed bypassing Goobi, so that Goobi was previously unaware of these renaming changes and this could lead to problems when re-exporting the METS files.
Setting the default export for METS files without considering the MIME type
With an adaptation of the standard export, we now circumvent this problem. Because in the case of the project configuration that the METS file should dynamically generate the file names depending on the recognised MIME type (e.g. image/jpeg), an update of the internally stored file names changed in the meantime is now also carried out within the METS file. In this way, such potential errors in the case of renaming carried out independently of Goobi (e.g. adjusted upper and lower case, changed file extension) are intercepted.
Setting the default export for METS files to consider the MIME type
Less spectacular for non-programmers sounds an elaborate change we are currently making internally for further code optimisations. We are currently using the Lombok program library even more for code generation of boring, bloated but unfortunately necessary source code, which by convention has to be within Java applications.
For those readers who are Java programmers themselves: Lombok is definitely worth a look and takes care of the automatic generation of getters, setters, constructors and a lot more.
Generation of getters and setters by Lombok
https://github.com/intranda/goobi-workflow/commit/17a0d32698c2c941cc7ec47317b08e0904dac756 https://github.com/intranda/goobi-workflow/commit/ef797453faf8c83a5bb6cd5096d04106cbebae18 https://github.com/intranda/goobi-workflow/commit/a3ee13e8559eceb29f059557fdf80bb093bf73fd https://github.com/intranda/goobi-workflow/commit/5eb5eecb3dfe7e4bc103cd061b64d2b13dc17384 https://github.com/intranda/goobi-workflow/commit/afd5b82169ed063da7586ed9c329c5465785336f https://github.com/intranda/goobi-workflow/commit/93b8d5cc45897e949322b79a8e3a7634bc1396c2 https://github.com/intranda/goobi-workflow/commit/16e903327281e247d6e3da84c4bf6224d89175f3 https://github.com/intranda/goobi-workflow/commit/665f4bd3e2d9b94d479f0fbdb89a0bea3f66d264
We have recently massively expanded the metadata groups. For a project in Croatia, we needed some rework here: It was necessary that metadata groups can also be duplicated, so that they now correctly duplicate all associated metadata, persons and subgroups. This saves some unnecessary manual editing.
Metadata groups can now be completely duplicated