General

With this last version of the year, we are probably releasing the last Goobi workflow release that can still be operated under Java 11 and Tomcat 9. From next month, we will begin the transition to Java 21 and Tomcat 10.

Attention: Please note that the upcoming releases will require a one-time major update effort.

Core

Process properties with vocabulary content

Process properties have existed in Goobi for a long time. They are often used to record and evaluate information during the workflow that is not part of the digitised material's metadata. Typical examples of this are specifications for working methods, scripts or statistics (e.g. opening angle, reading direction, glass plate utilisation).

Previously, the properties in the configuration file goobi_processProperties.xml could be typified for different content, for example to define free text fields, checkboxes or selection lists from which the user should select information.

What is new is that such information can now also come from vocabularies. They are defined in the same configuration file.

This means that the properties are now available in the modified interface with the values that were specified there as the main value.

Clicking on such a value opens a window that shows all the details of the vocabulary entry.

A data entry screen is also available for editing properties, which lists the values from the vocabularies.

Extension for handling hierarchical vocabularies

The major development work on the new vocabulary system has already taken place in recent months. In this context, vocabularies can also be structured hierarchically.

What was missing until now, however, was the ability to select vocabulary content within drop-down lists for properties. This is now possible with this update.

Further preparatory work for Goobi-to-go changeover

As already announced at the Goobi Days, we are currently working intensively on a complete overhaul of Goobi-to-go. This is currently being implemented and tested on the basis of Docker and already looks very promising. This month, we have adapted the start of Goobi workflow specifically for this purpose so that it is more robust in the event of a slower database connection of a freshly started database. In our opinion, this only plays a major role for Goobi-to-go on Docker as an infrastructure.

Plugins

Migration from Visual Library to Goobi

Time and again, users of Visual Library software approach us and plan to migrate their data to Goobi. We have now developed a plugin for this purpose that greatly simplifies the migration process. The data transfer from a running VL instance can be carried out in the same way as a catalogue query. Goobi then starts a workflow that uses the migration plugin to query the data via the OAI interface and, in addition to pagination, structural data and metadata, also automatically migrates the images and full texts to Goobi. An example of such a workflow is as follows:

Following this migration, the works will be available in Goobi as usual and can also be edited there in the metadata editor, for example:

Export to other systems such as long-term archiving, repositories and catalogues as well as publication in the Goobi viewer or other systems is also possible as usual:

A screencast is available here to illustrate how such a migration process can be carried out for individual works of different Visual Library instances:

Further information on the plugin for migrating Visual Library content to Goobi can be found in the documentation here:

Adjustments to the Transkribus plugin

Transkribus is very suitable for performing OCR or HCR and can also be automated very easily using Goobi with the existing plugin. In the past, when processing really large lists of processes with significantly more than 100,000 pages in particular, connections were often interrupted, resulting in time-consuming manual intervention. However, these are no longer necessary. A revision of the connection of Transkribus with an automatic repetition of connection attempts in the event that Transkribus cannot be reached now results in significantly less problematic mass processing of OCR. At the same time, the processing of the processes has also been adapted in such a way that they are now processed together and the overview of the progress has been significantly improved. A side effect of this was an enormous increase in page throughput.

More information on OCR processing using Transkribus can be found in the documentation here:

Extension of the validation methods for image files

The Tif validation plugin has been slightly extended to perform further validations on the basis of JHove. The comparison options have been extended so that the expected values can now be defined even more flexibly. The examples for the use of these comparison options in the reference configuration and also in the documentation have also been significantly expanded, so that initial installation of the plugin together with the documentation should now be much easier and more flexible. The updated documentation of the plugin can be found here:

Customisation of the newspaper export for the DDB newspaper profile

Goobi workflow and Goobi viewer already handle newspaper editions very elegantly and offer various options, e.g. with calendars, metadata and structural data, so that users can easily navigate to the desired newspapers. For the import of newspapers into the newspaper portal of the German Digital Library, however, a special processing of the metadata is necessary so that the newspaper, the year and the individual issues are displayed differently as intermediate levels. The necessary conventions have now been taken into account within a special export plug-in, so that the transfer to the DDB is now even easier.

Code analysis

Version number

The current version number of Goobi workflow with this release is: 24.12.4. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
    <groupId>io.goobi.workflow</groupId>
    <artifactId>workflow-core</artifactId>
    <version>24.12.4</version>
    <classifier>classes</classifier>
</dependency>

General information

As expected, the changeover to Java 21 has brought us some additional work. The changes related to both the infrastructure and the user interface, where we completed the Bootstrap 5 conversion at the same time. There is also a major innovation to report in the Goobi-to-go area.

Core

Java 21 - Rework here and there

We somehow saw it coming: There was still some rework to be done regarding the major changeover to Java 21. In particular, those plugins that were based on the Riot infrastructure required a few minor changes. Overall, however, these changes were absolutely manageable, so that the new infrastructure based on Java 21 and Tomcat 10 could already go live with some Goobi users.

Bootstrap 5 - Rework for the conversion of the metadata editor

There was also some minor reworking of the metadata editor. We had already converted it to Bootstrap 5 with the last release, so some areas of the metadata editor looked a little different. However, we have noticed a few minor optimisations, particularly with regard to usability, that still need to be fixed. For example, validation messages that should be displayed directly next to the fields have been slightly restyled:

The popover that appears when you move the mouse pointer over the structural elements in the left-hand area has also been customised with regard to the validation messages:

The seemingly little-known bookmark functionality has also been brought up to date in this context:

The status of the individual boxes, which users can expand and collapse in the structuring and metadata areas, has also been enhanced so that it is now retained when changing pages, thus avoiding unnecessary clicks.

Code cleanup of 200,000 lines

As we believe that the conversion to Bootstrap 5 is complete, we have decided this month to remove some of the legacy layout code. This applies in particular to HTML components, JavaScript files and styling specifications that were still present in various places for the use of Bootstrap 3 and Bootstrap 4 and which were intended to prevent areas that we may have inadvertently not yet converted from continuing to look good and remain usable. From our point of view, these files are no longer in use and have therefore been deleted. As a result, Goobi workflow has suddenly been reduced by around 200,000 lines of redundant source code. For us as developers, this feels good. :)

French introduced as a language

The popularity of Goobi worklow is apparently continuing to grow. There are always requests for additional languages to be supported. This month, we have therefore added French to the list of supported languages. For us as non-native speakers, this means that we provide the infrastructure for this, that there is a language switcher that now offers French as a language. We have also used automatic translation to ensure that French translations are now available for all labelling displayed in Goobi workflow, e.g. for buttons or fields. However, these have not yet been manually proofread by a native speaker. We are happy to provide interested parties with the language files for each of the languages already available so that they can complete or revise their translations.

Plugins

Conditions for the automatic updating of metadata fields

The plugin goobi-plugin-step-metadata-update-field has already existed for some time and is used to automatically fill metadata fields from the contents of other metadata fields and also generated information.

This plugin has now been extended to allow this generation of metadata and thus the update of the METS file of the corresponding process to only be executed if a configured condition is met. An example configuration for this looks like this:

<config_plugin>
    <config>
		<project>*</project>
		<step>*</step>
        
        <update>
			<field>TitleDocMain</field>
			<element>Monograph</element>
			<forceUpdate>true</forceUpdate>
            <condition field="PublicationYear" value="\d[4}]" />
    		<content type="variable">{meta.CatalogIDDigital}</content>
			<content type="static">_</content>
            <content type="metadata">PublicationYear</content>
		</update>

	</config>
</config_plugin>

More detailed information can be found in the plugin documentation, where the configurability of the conditions is also explained:

New plugin for METS file preparation

In some use cases, there is a need for the METS file of a process to be exported automatically without it having been edited manually by a user in the metadata editor. In such a case, it is usually necessary to enrich the METS file somewhat. In particular, the associated media files (e.g. the images) must be assigned to the METS file, at least at the top level. It may also make sense to automatically assign a pagination of different types or to automatically add other metadata. We have now developed a new plugin called goobi-plugin-step-mets-enhancer for such use cases.

As usual, the detailed documentation can be found on our documentation platform here:

The source code of the plugin and the installable version of the plugin can be found here:

Conversion of the rich text editor for transcriptions

The goobi-plugin-step-transcription plugin has been available for some time now as a relatively simple way of creating transcriptions of images and combining them with format instructions. So far we have used the editor TinyMCE for this purpose. For various reasons, however, we had to make a major change here in order to ensure the further development of the Goobi plugin. We have now decided to use the Quill JavaScript library for this purpose.

The new editor behaves in much the same way as the previous one. The update should therefore have no negative impact on users and work as usual.

The documentation of the plugin can be found here on our documentation platform:

The installable plugin can be found here:

New Goobi-to-go infrastructure

After a long period of work, we have finalised our infrastructure for Goobi-to-go. Initial tests with users are already taking place. What is new here is that the new Goobi-to-go infrastructure is based on Docker and is now much more flexible than before. It also finally runs on all common operating systems again.

We have not yet published the new version of Goobi-to-go independently. Anyone interested in carrying out tests is welcome to contact us at any time. We will then be happy to make this available.

Code analysis

Version number

The current version number of Goobi workflow with this release is: 25.03. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
    <groupId>io.goobi.workflow</groupId>
    <artifactId>workflow-core</artifactId>
    <version>25.03</version>
    <classifier>classes</classifier>
</dependency>

General information

This month we were finally able to finalise the major conversion of Goobi workflow and all plugins to Java 21. Due to the complexity, we expect that there will certainly be some rework here and there. However, the major changeover has finally been completed and we can now start installing the new versions on users' systems.

Core

Conversion to Java 21 and Tomcat 10

The migration of Goobi workflow core to Java 21 and Tomcat 10 has been completed and can be used on production systems from this release onwards. As already mentioned in previous digests, this changeover was necessary in order to be able to continue developing the software itself and its underlying infrastructure in the future and to provide it with security updates. Last but not least, the switch to Java 21 also took place against the background that the individual operating systems under which Goobi is operated are gradually discontinuing their support for Java 11.

Caution: Updating to this new version generally involves increased effort, as in addition to updating the actual Goobi application and the associated plug-ins, an operating system update, a Java update and a Tomcat update are also required.

Conversion of the metadata editor to Bootstrap 5

As has already been mentioned here several times, we have made a major change to the user interface in order to migrate the various areas of Goobi workflow and its plugins from Bootstrap 3 to Bootstrap 5. This is not only useful for a contemporary design of the user interface, but also to ensure that we meet the requirements for accessibility requirements and better usability.

One area that was previously excluded from the developments was the user interface of the metadata editor. Until recently, this was based on Bootstrap 4 and used a slightly different internal template for the display, layout and individual components (such as form fields, boxes, buttons, etc.). With the February 2025 release, we have now removed this different metadata editor infrastructure and also migrated the metadata editor to Bootstrap 5. As a result, the user interface now looks slightly different in some areas. Functionally, however, everything should remain the same as before. What users will probably notice most is that the form fields are now designed in a similar way to the forms that are usually found in Goobi workflow:

The arrangement of the standard data fields, e.g. for persons and places, looks a little tidier than before:

The recording of new structural elements has been slightly revised:

Validation messages are now also displayed in a slightly different form:

Overall, we hope that this change will not only result in a somewhat more standardised user interface with better usability and accessibility, but will also make it easier to further develop the metadata editor in future.

Refactorisation for the frontend

We have also made some internal changes to the structure of the HTML pages to simplify development. The template based on JavaServer Faces was reorganised so that additional facelet tags are used. Repeated blocks in the user interface are thus displayed more uniformly and are easier for us as developers to control from a central location. This is also visible to users in the revised properties area:

Plugins

New plugin for compiling batches

The functionality of batches is only really used by relatively few users on a daily basis. However, some organisations already work primarily with batches in very large projects in order to run many processes together through their workflows. Previously, it was necessary to either create the batches together when the processes were created and thus combine them into one batch. Or batches had to be compiled in the batch interface in a somewhat cumbersome way.

With a new plugin, we are now trying to make it easier for other users to work with batches. To this end, a new step plugin has been implemented that enables users within a task to add the respective process to an existing (and still waiting) batch and also to mark this batch as complete so that its further processing can begin.

Alternatively, it is also possible here to create a new batch together with a title and configurable properties, which can also receive other processes for batch processing in the further course.

Detailed documentation of the plugin can be found as usual on our documentation platform:

New plugin for convenient handling of barcodes

Goobi workflow has been able to handle barcodes well since the very first versions. The generation of barcodes on dockets, a barcode generation plugin, a mass upload with barcode analysis on images and, of course, searching using barcode readers have long been possible.

What was still missing, however, was extensive automation of various actions simply by scanning barcodes. We have developed a completely new plugin for this purpose that can be opened at any time from the Goobi worflow interface without leaving the current interface.

After clicking on the corresponding icon, it opens immediately as a modal above the current interface.

The possibilities here are almost unlimited. For example, stored actions can first be selected by barcode scans (e.g. setting a property, changing a step status, performing an export, manipulating workflows). The selected action can then be executed for the relevant processes with each scan of a process barcode or batch barcode. The elegant thing here is that the actions that can be used are all those that are also available via GoobiScript.

The actions to be used here can be defined very granularly in the plugin's configuration file. For convenient use of the functionality, it is then possible to generate barcodes for the stored actions directly from the plugin so that they can then be printed out, for example.

To automatically track the location of an object, it is therefore very easy, for example, to generate barcodes for individual offices, which can then be used for each object when it arrives at an office with a barcode reader and thus carry the current location of the object as a process property in Goobi.

Documentation for this new plugin has not yet been created and is not yet online. This is expected to be published next month.

Internal changes for handling Riot-based plugins

Due to the switch to Java 21, all plugins that were technically based on the Riot infrastructure and communicated between the frontend and backend via REST had to undergo a major change to the REST connection. As a result, the access rights for some REST endpoints may have to be adjusted after the update to the version described here. Details of the adjustments can always be found in the Goobi workflow update instructions:

Code analysis

Version number

The current version number of Goobi workflow with this release is: 25.02. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
    <groupId>io.goobi.workflow</groupId>
    <artifactId>workflow-core</artifactId>
    <version>25.02</version>
    <classifier>classes</classifier>
</dependency>

General

As in previous years, the big user meeting was held in Göttingen again this year. Nevertheless, we were still able to achieve a lot this month.

Core

Conversion to Java 17

We have changed the source code of Goobi and the plugins from Java 11 to Java 17. This is not a major change for the time being because we are still compiling the source code with compatibility for Java 11. This means that the application can still be used in the tried and tested infrastructure without having to make any major changes. However, we will soon have to carry out a major update for Java 21 in order to be compatible with Tomcat 10. We will tackle this at the beginning of next year

New interface for the journal

We have completely redesigned the interface of the journal, which is part of every process. Instead of working with different colours as before, a classification of the individual messages is now recognisable by the icons. This allows for much better accessibility.

As before, the journal can be switched to an expanded mode and is therefore much easier to read, especially with a lot of content, without having to scroll within the box.

And, of course, we have designed it in such a way that all areas of Goobi worklfow that use the journal benefit from it and therefore utilise the same internal components and therefore the same design. This applies, among other things, to the journal in the accepted tasks, in the projects and among the users.

Plugins

Processing user orders

Together with the Zentralbibliothek Zürich, we have created a very flexible plugin for handling user orders. This allows information provided by users when placing an order to be recorded and processed for billing purposes. On the one hand, this involves the possibility for users to be informed about their orders and to receive a delivery note.

However, it was also important to ensure that order details that have an influence on the composition of costs can be processed and then made available to the accounting department.

Extension of the mail plugin for attachments and variables

For some time now, there has been a plugin that allows Goobi to send e-mails with flexible content during the workflow. What was missing, however, was the ability to use information from the process properties to specify recipients. This was necessary for sending messages to the senders of a user order.

In this context, the plugin has also been extended to allow attachments to be added to emails. These can also be defined very flexibly using variables from properties, metadata or paths of the process.

PDF generation extended

The plugin, which can be used to flexibly generate PDF files in the course of workflows, has been significantly revised. The configuration of the plugin now allows even better control of where the PDF files should actually be generated from. On the one hand, this affects the specification of the directory from which the image files are to be used for the PDF files. On the other hand, it also affects the control of how the PDF is to be assembled. You can now specify whether individual pages are to be generated as PDFs or whether an overall PDF is to be generated. You can also control how the composition is to take place, so that, for example, an overall PDF is to be assembled from individual pages or controlled by the page assignment of the METS file. The location to which the PDF files are to be exported can now also be granularly controlled.

Extensive documentation work

We are currently working very intensively on developing the documentation. On the one hand, this includes maintaining the actual documentation and configuration, including installation instructions in German and English. However, it also includes explaining the plugins in digests whenever there have been relevant changes, publishing them on Github and ensuring that they have an up-to-date readme file that guarantees a link between the reader-friendly documentation and the source code.

For the following plugins, there were either extensive updates to the documentation or new documentation was created:

There have been changes to the README files for the following plugins this month:

goobi-plugin-administration-data-poller
goobi-plugin-administration-goobi2goobi-import
goobi-plugin-administration-reset-pagination
goobi-plugin-administration-restore-archived-image-folders
goobi-plugin-administration-ruleset-compatibility
goobi-plugin-dashboard-barcode
goobi-plugin-dashboard-extended
goobi-plugin-export-adm-bsme
goobi-plugin-export-bda-single-image
goobi-plugin-export-configurable
goobi-plugin-export-fedora
goobi-plugin-export-fedora-prov
goobi-plugin-export-heris
goobi-plugin-export-newspaper
goobi-plugin-export-nli-pdf-to-folder-structure
goobi-plugin-export-selected-images
goobi-plugin-export-stanford
goobi-plugin-export-vlm
goobi-plugin-export-weimar-haab
goobi-plugin-export-zop
goobi-plugin-import-bka-bda
goobi-plugin-import-crown
goobi-plugin-import-eth-no-catalogue
goobi-plugin-import-excel
goobi-plugin-import-katzoom
goobi-plugin-import-mab
goobi-plugin-import-sisis-sunrise-file
goobi-plugin-import-wuwien
goobi-plugin-import-zbz-alma
goobi-plugin-import-zbz-cmi
goobi-plugin-import-zbz-no-catalogue
goobi-plugin-metadata-change-type
goobi-plugin-metadata-create-structure-elements
goobi-plugin-opac-ariadne
goobi-plugin-quartz-heris
goobi-plugin-statistics-sudan-memory
goobi-plugin-statistics-user-throughput
goobi-plugin-opac-ead
goobi-plugin-opac-generic-xml
goobi-plugin-opac-json
goobi-plugin-opac-kalliope
goobi-plugin-opac-marc
goobi-plugin-opac-pica
goobi-plugin-opac-soutron
goobi-plugin-quartz-bka-wohnbau
goobi-plugin-step-abbyy-finereader
goobi-plugin-step-alma-api
goobi-plugin-step-analysis-imagename
goobi-plugin-step-archive-image-folder
goobi-plugin-step-ark
goobi-plugin-step-bag-creation
goobi-plugin-step-batch-progress
goobi-plugin-step-catalogue-request
goobi-plugin-step-change-workflow
goobi-plugin-step-create-full-pdf
goobi-plugin-step-create-structure-from-filenames
goobi-plugin-step-datacite-doi
goobi-plugin-step-delay
goobi-plugin-step-delay-workflow-status
goobi-plugin-step-delete-content
goobi-plugin-step-display-metadata
goobi-plugin-step-doi
goobi-plugin-step-download-and-verify-assets
goobi-plugin-step-duplicate-tasks
goobi-plugin-step-epic-pid
goobi-plugin-step-ewig
goobi-plugin-step-excel-metadata-enrichment
goobi-plugin-step-export-package
goobi-plugin-step-fetch-images-from-metadata
goobi-plugin-step-file-upload
goobi-plugin-step-file-validation
goobi-plugin-step-flex-editor
goobi-plugin-step-generate-alto-ids
goobi-plugin-step-generate-identifier
goobi-plugin-step-geonames-auto-annotator
goobi-plugin-step-geonames-correction
goobi-plugin-step-handle-mets
goobi-plugin-step-heris
goobi-plugin-step-image-metadata-extraction
goobi-plugin-step-image-resize-and-watermark
goobi-plugin-step-image-selection
goobi-plugin-step-imageqa
goobi-plugin-step-import-from-folder
goobi-plugin-step-kiel-archive-cleanup
goobi-plugin-step-metadata-edition
goobi-plugin-step-metadata-per-image
goobi-plugin-step-metadata-replacement
goobi-plugin-step-metadata-structure-import
goobi-plugin-step-metadata-update-field
goobi-plugin-step-mets-creation
goobi-plugin-step-mets-enhancer
goobi-plugin-step-mix-metadata-enrichment
goobi-plugin-step-mixed-ocr
goobi-plugin-step-ocr-selector
goobi-plugin-step-ocr-to-metadata
goobi-plugin-step-oid-creation
goobi-plugin-step-olr-correction
goobi-plugin-step-pdf-book-interchange-convert
goobi-plugin-step-pdf-extraction
goobi-plugin-step-pdf-upload
goobi-plugin-step-placeholder-creation
goobi-plugin-step-process-folder-migration
goobi-plugin-step-rename-files
goobi-plugin-step-rename-files-before-rosetta
goobi-plugin-step-rename-process
goobi-plugin-step-reorder-images
goobi-plugin-step-replace-images
goobi-plugin-step-representative-creation
goobi-plugin-step-reverse-image-order
goobi-plugin-step-rki-import
goobi-plugin-step-save-docket-file
goobi-plugin-step-send-mail
goobi-plugin-step-tei-import
goobi-plugin-step-tif-validation
goobi-plugin-step-transcription
goobi-plugin-step-transkribus-collection
goobi-plugin-step-urn
goobi-plugin-step-user-assignment
goobi-plugin-step-vocabulary-enrichment
goobi-plugin-step-xmp-header
goobi-plugin-step-yerusha-validation
goobi-plugin-step-zbz-order-delivery
goobi-plugin-workflow-aeon-process-creation
goobi-plugin-workflow-barcode-generator
goobi-plugin-workflow-charite-importer
goobi-plugin-workflow-excel-import
goobi-plugin-workflow-file-upload-process-creation
goobi-plugin-workflow-hu-importer
goobi-plugin-workflow-import-json
goobi-plugin-workflow-layoutwizzard
goobi-plugin-workflow-linked-process-creation
goobi-plugin-workflow-mass-upload
goobi-plugin-workflow-newspaper-pages-importer
goobi-plugin-workflow-project-export
goobi-plugin-workflow-yerusha-aim25

Version number

The current version number of Goobi workflow with this release is: 24.09.9. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
    <groupId>io.goobi.workflow</groupId>
    <artifactId>workflow-core</artifactId>
    <version>24.09.9</version>
    <classifier>classes</classifier>
</dependency>

General

This month, the focus was mainly on implementing the new vocabulary management system. However, the archive management has also been given a new look.

Core

More flexible folder configuration

For some time now, it has been possible to configure various folders in order to use other directory structures in addition to master and media. This functionality has now been extended so that the configurable folders can now also be configured for other locations and are not just limited to those within the images directory of the process. An example configuration looks like this:

The directories are defined in the configuration file goobi_config.properties like this:

process.folder.ocr.pdf={processtitle}_pdf
process.folder.export.files={processtitle}_pdf
process.folder.validation.jpylyzer={processtitle}_xml

Such defined paths can then be used within a plugin, for example in the FileUpload plugin in its configuration file plugin_intranda_step_fileUpload.xml:

<config_plugin>
    <config>
            <project>*</project>
            <step>*</step>
            <regex>/.*/</regex>
            <folder>master</folder>
            <folder>media</folder>
            <folder>ocr.pdf</folder>
            <folder>export.files</folder>
            <folder>validation.jpylyzer</folder>                        
    </config>
</config_plugin>

Vocabulary

Work on the vocabulary server started last month and is continuing. There is not much visible to report yet. We expect to be able to report more detailed progress in the coming weeks.

Plugins

Archive management

For greater performance and flexibility in the overall EAD / archive context, we have decided to remove the storage of archive records from the separately installed BaseX database and instead store all EAD data internally in the Goobi database, managed by Goobi workflow. This under-the-hood work has now been largely completed. This has already resulted in an enormous increase in performance.

The first changes to the user interface have now also been made. For example, it has been implemented that the NodeID can now always be displayed in the archive tree and nodes can be created in bulk and provided with metadata. The display of the metadata of the individual nodes can also be controlled somewhat better via the configuration file and links can be created between different nodes.

We will be busy with the developments for a few more weeks. We will report back here.

Version number

The current version number of Goobi workflow with this release is: 24.06.1. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
    <groupId>io.goobi.workflow</groupId>
    <artifactId>workflow-core</artifactId>
    <version>24.06.1</version>
    <classifier>classes</classifier>
</dependency>

General

This month, the focus was on three things in particular: remodelling the vocabulary, implementing the new archive management system and developing the plugin for the Austrian housing fund.

Core

Vocabulary

Development work on the new vocabulary server is progressing well. Functionally, everything has been prepared for the new vocabulary server to such an extent that we were able to concentrate on implementing the user interface this month.

The aim is to make as much as possible easy to use via the user interface. We will not yet implement an administrative interface for creating new vocabularies in this context.

User interface and Javascript frontend build

There are a lot of legacy issues that have accumulated over the years that we are currently cleaning up in parallel. We have been working on a revision of the user interface for quite some time now. In this context, we are also checking the dependencies of the JavaScript libraries used in the background as well as the entire compilation process for the user interface. Much of this work is not directly visible in the application later on. However, this work is still necessary.

Plugins

Data import for the Austrian housing fund

An automated data import was developed for the Austrian housing fund. This involves digitised files, which are provided in the form of several deliveries by the scanning service provider and are stored in json files, being broken down into their respective documents, structured and enriched with metadata.

The plugin is very specific to the Austrian housing fund and is therefore hardly usable for other projects. Nevertheless, further details can be found here on the documentation page for those interested:

Implementation of invoice generation and delivery note generation with cost recording

The central library in Zurich needed to be able to process user orders as automatically as possible. A very special workflow was developed for this purpose, which analyses the user orders and carries out the desired Workflow steps accordingly.

One part of the workflow is that information on costs can also be recorded. These are dependent on the currency, the order, the delivery type and other factors, so that a manual adjustment of the cost breakdown is actually required.

Finally, Goobi uses this plugin to automatically generate delivery notes, upload the data to a target server and also generate an invoice template for the accounting department.

More details about the plugin can be found as usual here on the documentation page:

Archive management

We are in the final stages of the major remodelling of the archives. Our planning envisages that we will be finished with the developments next month. Detailed information about the new features will then finally be available.

Version number

The current version number of Goobi workflow with this release is: 24.07. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
    <groupId>io.goobi.workflow</groupId>
    <artifactId>workflow-core</artifactId>
    <version>24.07</version>
    <classifier>classes</classifier>
</dependency>

General

Changes at the Maven structure

We implemented the Maven infrastructure change in April. Goobi workflow still looks the same, but a lot has changed under the bonnet. This is particularly interesting for developers and system administrators.

The new structure now makes it easier to refactor and maintain dependencies in a central location.

Note for developers

For developers, there is now no longer the goobi-workflow repository alone, but workflow-base and goobi-workflow-core. We recommend that developers delete all local repositories and check them out again from Github. The core and the plugins now define workflow-base as parent. The location of the artefacts in Nexus has also changed. They are no longer located under de.intranda.goobi.workflow, but under io.goobi.workflow.

Note for system administrators

For system administrators, the naming of artefacts in the plugins has changed. The spelling of .jar plugin files has been consistently harmonised and standardised. Configuration files and the names of the plugins have not been renamed. Nevertheless, care should be taken in future updates to ensure that no old artefacts are left behind.

Snippets

• A larger set of special plugins was finalised for a customer

• Work in the context of vocabularies and archive is progressing

Version number

The current version number of Goobi workflow with this release is: 24.04.30. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
    <groupId>io.goobi.workflow</groupId>
    <artifactId>workflow-core</artifactId>
    <version>24.04.30</version>
    <classifier>classes</classifier>
</dependency>

General

Core

Config Editor

The former plugin for the Goobi workflow configuration editor has been integrated into the core and is available from this release without additional installation.

Frontend build system

The interface of Goobi workflow is significantly styled using the classic web technologies CSS and Javascript. However, development takes place in SASS or LESS files, for example, which are then compiled into CSS. There are also various individual files in JavaScript, which are then combined into one file for the application. In the past, compiling and summarising took place on the developer's PC, who then checked the finished files into the repository. From the February release onwards, these steps will be carried out automatically in the CI environment. This simplifies the development process and reduces the potential for merge conflicts.

Snippets

• The comparator for file names has been fundamentally revised so that ascending and descending sorting of files in the core and in plugins is easier and more error-free.

Plugins

Export: VLM

The VLM export plugin can now use different configuration blocks based on metadata and conditions.

Step: Rename files

The step plugin for renaming files has been massively extended and can now contain several <condition /> and <replace /> arguments in the new configuration section <namepart />.

Step: Metadata edition

There was a bugfix in the step plugin for editing metadata to save information about the representative correctly.

Code analysis

Version number

The current version number of Goobi workflow with this release is: 24.02.2. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>24.02.2</version>
</dependency>

General

This month's release comes a little late and is correspondingly extensive. We will limit ourselves here to the most important points of this month.

Core

Access restrictions for metadata

The UGH program library, which as part of Goobi is responsible for ensuring that metadata, structural data, pagination, catalogue mapping, exports and much more function properly, has been extended so that individual metadata can now be assigned access restrictions. Within the rule set, an additional attribute allowAccessRestriction must simply be added to the desired metadata.

Once this configuration has been made in the rule set, the metadata editor displays a small icon in its interface in front of the respective metadata that allows the access restriction to be activated.

If works have been assigned such defined access restrictions and the exported METS file therefore contains the access restriction information, a target system such as the Goobi viewer can make use of this. In the case of the Goobi viewer, for example, the METS file is now checked for such restrictions for access by users and then cleansed of the protected data for unauthorised persons. In this way, the information is not only hidden in the interface for unauthorised persons but also for access to the underlying METS file.

Docker support extended - preparation for a new Goobi-to-go

There are already some Goobi users who no longer install Goobi workflow directly on the servers and instead use Docker to run the application within containers. A Docker image has therefore been available for Goobi workflow for some time, which can be found here:

We are currently working relatively intensively on this infrastructure and are converting some of it. Not only so that users who want to install it on their servers can benefit from it. But also because we have been working for some time on designing a completely new version of the popular Goobi-to-go, which will then be operated on the basis of Docker. So there are big changes and announcements to come in the next few weeks.

Process properties with better labelling

Previously, process properties were displayed in a common box and listed one below the other. In the past, it usually looked something like this:

The individual areas could group several properties as containers. But there was no obvious classification of these containers. These are now to be extended so that they are no longer just listed, but can actually be displayed as independent containers, where the affiliation to the respective containers is also more obvious:

To make operation even easier for users, however, container management has now been extended so that each container can now be given a name. This is defined in the configuration file as follows, for example:

The presentation of the properties is then clearer for the user and the affiliation to the respective named containers is easier to understand:

The same presentation in this box form is of course also taken into account within the assumed tasks in order to ensure standardised operation:

New notification system and footer

We already announced it at the Goobi Days. Now it has also gone into production: We have implemented a completely different display of notifications. Success and error messages are no longer simply displayed at the top, but can now also be hidden and superimposed on the interface so that they no longer interfere so much with the layout of the entire page if several messages are displayed at the same time.

The original plan was that the messages might also come from the footer area of Goobi. We have therefore equipped Goobi worklow with a footer that already contains some useful links and information. We will be adding more practical things here in the future.

Vocabulary server as a Docker image

So that the new vocabulary server can not only be installed regularly on Severn, we have now also made it available as a Docker image. This also makes it easier to operate within Docker environments. In the future, this will also enable us to make good use of the vocabulary server within the new Goobi-to-go infrastructure.

Vocabulary with monitoring interface

The vocabulary server has been made a little more robust. Among other things, it has a method for checking its own operation and makes this available as a query for monitoring. This makes it possible for external applications such as Goobi workflow to check the correct operation of the vocabulary server before communicating with it. This makes it easier to avoid potential errors.

A version check was also implemented in this context, which allows other applications to query the version status of the vocabulary server. Based on this, external applications can decide better whether and how they want to continue communicating with the vocabulary server.

User interface based on Bootstrap 5

We are of course still working on the Goobi workflow user interface. As announced, the switch to Bootstrap 5 is in full swing and unfortunately still not complete. Overall, however, we have already made great progress and are now focussing on revising the user interface of the 300 or so plug-ins so that Goobi workflow and the plug-ins have a uniform appearance.

Plugins

Revision of the entity editor

The editor for editing entities has been significantly revised this month in order to simplify operation, improve the storage of relationships between data records, performance and interaction with the vocabulary server.

Revision of the Data Poller

During the Goobi Days in September, it was once again discussed that the functionality of the Data Poller (formerly Catalogue Poller) needs to be intensively reviewed. In the previous months, this had been equipped with the option of test runs, the results of which should be visible to users before they are actually executed. There was also a desire for a configurable whitelist for the metadata to be updated instead of the previously only available blacklist for the metadata not to be updated.

The functionality has now been tested and errors have been corrected. In this context, the user interface has also been ported to Bootstrap 5.

Version number

The current version number of Goobi workflow with this release is: 24.10. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
    <groupId>io.goobi.workflow</groupId>
    <artifactId>workflow-core</artifactId>
    <version>24.10</version>
    <classifier>classes</classifier>
</dependency>

General

The changes to the Maven infrastructure described in April have taken much longer than expected. However, progress has also been made in other areas, for example with the vocabularies and some plugins.

Core

ALTO Editor

A new tagging functionality has been added to the solution integrated into the metadata editor for editing ALTO files. One or more words can be marked and linked to an entity type and standard data. The standard data that is already known in the METS file is available in a drop-down menu. Alternatively, you can also enter your own URLs to enable the link to the standard data.

New development of vocabularies

Due to new requirements, it has become necessary to significantly revise the management of vocabularies. We have decided to manage the vocabularies in a separate vocabulary server in future. This will then be available to Goobi workflow, Goobi viewer and other solutions via an API in order to retrieve information from it. Development is still at a very early stage and will probably continue for a few more weeks.

Plugins

Connecting COPE to Goobi workflow

In one project, the need arose to connect the COPE software from Digital Transitions to Goobi workflow. It is used to convert RAW files from Capture One into TIFF files while retaining all relevant image information. The particular difficulty here was connecting this Windows-based software to Goobi. Using Python scripts, we have now succeeded and gone live.

Connection of the Transkribus Collections

In the context of a project, it became necessary to implement a further Transkribus connection. The specific aim was to use already digitised image files for labelling in Transkribus in order to be able to generate a ground truth for new recognition models.

Two new plugins were therefore implemented to realise this functionality. The first plugin is responsible for reporting the METS file already published in the Goobi viewer with the associated image files to Transkribus and importing them into the user's desired collection there.

The second plugin, on the other hand, is responsible for retrieving the data enriched in Transkribus and the results generated as ALTO files and saving them in the OCR folder of the respective process.

The documentation for both plugins can be found here:

The plugin can be downloaded for installation here:

Version number

The current version number of Goobi workflow with this release is: 24.05. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
    <groupId>io.goobi.workflow</groupId>
    <artifactId>workflow-core</artifactId>
    <version>24.05</version>
    <classifier>classes</classifier>
</dependency>

Core

Tables

Finally, the column width can be customised within Goobi workflow. The information is saved in the browser's local storage. Once a setting has been made, it is therefore linked to the PC and the browser and is not stored on the server side. This means that different settings can be made on different PCs with different screens so that they do not always get in each other's way.

GoobiScript

If there are several export steps in a process, you can now pass the title in the GoobiScript so that the correct export is triggered. Example:

---
# This GoobiScript allows to export Goobi processes using the default export mechanism. It either uses the default export or alternativly an export plugin that was configured in one of the workflow steps.
action: export

# Decide if the images shall get exported additionally to the metdata (`true`).
exportImages: false

# Decide if the OCR results shall get exported additionally as well (`true`).
exportOcr: false

# Decide which export plugin shall be used by giving the name of this export plugin. It has the highest priority if it is configured. [OPTIONAL]
pluginName: 

# Decide which export plugin shall be used by giving the name of the export step. If it is configured then it will try to find this export step first, and only when there is no such step, will the standard choice be used instead. [OPTIONAL]
stepName: "Export into the Goobi viewer"

Snippets

• Various minor styling adjustments were made in the area of breadcrumbs or menus Initial internal preparations have been made for Java 17 support

• The message queue has received performance improvements

• A discontinued Java library for processing regular expressions that has not been maintained for many years has been removed and replaced by an alternative solution.

Plugins

Step: Downloading and verifying files

A step plugin has been developed to download files, verify them and then provide feedback to the external system.

Version number

The current version number of Goobi workflow with this release is: 23.12.3. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.12.3</version>
</dependency>

General

Core

User settings

For some time now, the number of images within a process can be displayed as an additional table column. Until now, however, it was not possible to activate this setting in general, but it had to be reset after each login. From the January release, it will be possible to save the option in the user settings so that it is active by default for an account.

Search

It is now possible to search for all tasks that have a task with a specific title. Previously, the status of the task had to be specified (steplocked:, stepopen:, stepinwork:, stepdone:, ...). From now on, simply step: followed by a title or part of a title also works.

Harvester

The harvester introduced in October has been extended to include a test mode for the OAI-PMH type. If this is activated, only the first page is queried rather than the complete hit set. The resumptionToken is ignored.

Static from and until values can also be set. If the values are empty, the system searches incrementally for all new data since the last query.

Accessibility

In the January release, we worked on many small construction sites in the context of improved usability and accessibility. For example, some page titles have been added, operation via the keyboard under Firefox has been improved, alternative texts for graphic elements have been added and the page structure has been improved. The changes are only noticeable when Goobi workflow is operated with a screen reader or without a mouse.

Security

For some time now, the journal has also been displayed for user accounts. From the January release, a message is automatically left there when the user profile is entered by an administrator.

Snippets

• The steptitle option is now optional in the GoobiScript setPriority function.

• An error that occurred during an empty search for authority data is now intercepted.

Plugins

Import: Excel

The generic import plugin for Excel files can now also process the creation of processes via GoobiScript.

OPAC: Soutron

The Soutron OPAC plugin now also supports the new XML API.

Code analysis

Version number

The current version number of Goobi workflow with this release is: 24.01.2. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>24.01.2</version>
</dependency>

Core

Harvester

Much of the functionality of the intranda harvester has been integrated into the Goobi workflow core. Repositories of different types can now be created and configured. The harvester queries a repository, for example of type OAI-PMH, at a specified frequency and downloads the data records to a specified folder. Repositories can be switched on or off individually.

It is possible to define a script that is to be applied to each downloaded data record. This makes it very easy to modify the downloaded files.

If desired, processes can also be created in Goobi workflow. To do this, a project, a production template and the import format must be selected. The mapping from the rule set is available for the import formats.

This integration allows data records to be harvested from Goobi workflow, for example, prepared via a script and then forwarded directly to the Goobi viewer hotfolder. Another scenario is that, for example, MARC-XML data records are downloaded from an OAI-PMH interface and processes are created in Goobi workflow for further work.

LIDO

The Java library UGH has been extended so that it can also generate empty elements, as is necessary for the LIDO format. With the following export mapping, for example, an element is created even if there is no value:

        <Metadata>
          <InternalName>Person</InternalName>
          <WriteXPath>./lido:eventActor/lido:displayActorInRole</WriteXPath>
          <exportEmptyNode>false</exportEmptyNode>
        </Metadata>

Plugins

Dashboard: Barcode

The dashboard plugin allows you to scan a barcode into an input box and directly execute a previously selected action. For example, a task can be accepted or accepted and directly completed again.

Import: Liechtensteiner Volksblatt

A special import plugin has been developed for importing the Liechtensteiner Volksblatt.

Code analysis

Version number

The current version number of Goobi workflow with this release is: 23.10.2. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.10.2</version>
</dependency>

Core

Process properties

The process properties available in Goobi workflow can now also be displayed as a text area. They are configured using the familiar goobi_processProperties.xml. An example configuration is as follows:

    <property name="Beschreibung" container="0">
        <project>*</project>
        <showProcessCreation access="write" template="*" />
        <type>Textarea</type>
        <defaultvalue></defaultvalue>
    </property>

Snippets

• If the GoobiScript for deleting processes was called several times in succession using the same hit set, an error occurred which made it necessary to restart the application. The behaviour has been corrected.

• The default configuration of ActiveMQ has been adjusted so that the service only listens on localhost.

• An error that occurred when repeating certain variables in script steps has been fixed

Plugins

Import: Charité - Female doctors in the German Empire and persecuted doctors

A special import plugin was developed for the import of the two databases "Female Doctors in the German Empire and the Weimar Republic" and "Persecuted Doctors of the Berlin Municipal Health Service (1933-1945)" and the conversion into METS/MODS files.

Step: Metadata per image

A very powerful step plugin was finalised this month. This allows a wide range of metadata and links to be recorded for each image in a process.

Step: Duplicate tasks

The plugin makes it possible to read a task property and duplicate an existing workflow step multiple times for each line in the property. It also duplicates the properties split into individual properties per line.

Step: Extract PDF files

The already known plugin for the fully automatic preparation of PDF files has been supplemented by free file naming and updated documentation.

Code analysis

Version number

The current version number of Goobi workflow with this release is: 23.11.1. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.11.1</version>
</dependency>

At this point we provide a detailed insight into our current developments. We try to keep them relatively up to date and to summarize for each month if possible what innovations or changes have been made.

In the table of contents on the left you will find a list of the respective months for which such a compilation already exists. And here you find a direct link to the latest digests:

General

Core

Config Editor

The newly integrated configuration editor has been extended to allow complete files to be uploaded and downloaded. Furthermore, the backups of the files written in the background are optionally offered for download in the user interface

Periodic tasks

There have been minor changes to the user interface in the Quartz job management section. When a task is paused, the schedule and date for the next execution are now not displayed. In addition, the icon for one-time execution has been changed.

Vocabularies

In the area of vocabularies, work has begun on standardising the usability of importing and exporting data records.

Archives

In the area of archives, work has begun on implementing the new Java logic for searching in collections.

Plugins

Export: ADM - BSME

An export plugin has been developed that supports the creation of special export formats in the Imagen Media Archive Management software.

Step: ALMA API

The ALMA API plugin can now also contain conditions for variables.

Step: Metadata enrichment via Excel file

The new step plugin allows metadata to be read from an Excel file and added to existing structural elements.

General

As the October release was published relatively late, there was not much time left in November to develop new features. The digest is correspondingly shorter.

Core

Adjustments to the vocabulary server

The possible uses of the new vocabulary server are much more extensive than we had initially assumed. And with these additional application options, there are always small adjustments that need to be made. For example, there was a need to revise the saving of vocabulary data records. However, the user interface has not changed for users.

There was also a need for Goobi workflow to have a check routine for interaction with the vocabulary server in order to query the version status of the vocabulary server and check a minimum version to ensure smooth communication between the two applications.

Swiss German as an additional language for the user interface

Even if Goobi workflow is already equipped with the German language in the user interface, this is not automatically the best choice for users from different German-speaking regions. For example, it has been found that the number formats make a big difference for users when it comes to billing for user orders.

The mere fact that the numeric keypad on a Swiss keyboard does not have a comma but a full stop made it somewhat awkward for Swiss users to enter numbers with decimal places.

The display of price information, which must be processed as part of the billing of user presentations, therefore also made it necessary for Goobi workflow to now offer Swiss German (DE-CH) in addition to German (DE) within the language switcher.

Plugins

Mass import for the advertising material archive

A new mass import plugin has been developed for the Vienna University of Economics and Business, which allows the advertising material and brand studies available there to be imported. These were previously available as PDF files and were described within JSON files. We have developed a new plugin to import them and carried out the import.

The detailed documentation for the plugin can be found here:

Code analysis

Version number

The current version number of Goobi workflow with this release is: 24.11.4. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

General

A lot of development work has taken place this month to migrate Goobi workflow and all plugins to Java 21. However, as the work is so extensive, we have decided not to publish a release this month. The latest release is therefore, for once, the one from the previous month.

Core

Conversion to Java 21 and Tomcat 10

Goobi workflow is currently based on Java 11 and can be operated under various operating systems with Apache Tomcat 9 as the web server. However, this technical basis is currently undergoing major changes. Java 11 will only be supported by the individual operating systems for a limited time. And Tomcat 9 is also no longer available in the repositories of several Linux distributions. Using Ubuntu Linux as an example, it is easy to see which versions are still supported and for how long. This in turn determines how long Java 11 will be available for the respective distributions:

Before support for Java 11 and Tomcat 9 expires for the typical systems on which Goobi is operated, we need to upgrade Goobi workflow and all associated components to a new version of Java and Tomcat. However, this is a very extensive task with regard to Java 21 and Tomcat 10. We are currently working on migrating around 400 repositories with source code to Java 21. We had hoped that we would be able to complete the work in January. However, the scope of the work is much larger than anticipated. Therefore, this can only remain an announcement for the upcoming release in February. We are still working on it.

Conversion of the metadata editor to Bootstrap 5

The conversion of the metadata editor to Bootstrap 5 is still in full swing, as it was previously based on Bootstrap 4 and was always slightly different in terms of operation and design to other areas of Goobi workflow. We are currently trying to eliminate these differences and are porting the metadata editor to the same HTML infrastructure as the rest of Goobi workflow. On the one hand, we hope that this will improve usability and accessibility. On the other hand, we also hope that these changes will make the future development of the metadata editor a little easier.

The work has not yet been finalised. However, here is a preview of what the interface will look like after the changeover:

Adaptations for mobile devices

With the increasing use of mobile devices such as tablets and mobile phones, Goobi workflow must of course also be easy to use. Here, too, there have been some minor adjustments to make operation easier.

Even the metadata editor is now easier to use on mobile devices. However, only time will tell whether users will make use of this.

Plugins

Rule set editor

For some time now, the rule set editor plug-in has provided Goobi workflow users with a convenient mechanism for configuring metadata and structural data directly from the user interface.

This editor not only allows convenient editing, but also the automatic creation of backups of the previous version when a change is saved. It also allows the entire rule set to be validated.

We worked intensively on this validation in December and January in order to expand it. It is great that numerous misconfigurations can now be checked and thus avoided. What is not so nice is that due to the complexity of the rule sets, which also take over the mapping of data from catalogues as MARC and PICA and into the target formats METS and LIDO, every rule set actually has errors that should be looked at.

Configurations that have grown historically, that appeared to be correct and have been changed over time through further adjustments, contain various small and non-critical but also critical errors that are now becoming apparent for the first time with the new validation options. It is therefore worth installing or updating this plugin to check your own rule sets thoroughly.

Code analysis

Version number

The current version number of Goobi workflow with this release is: 24.12.4. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

General

The new vocabulary server is finally ready and has already gone live. But many other areas of Goobi have also received new features this month.

Core

New vocabulary server

Goobi workflow has already had a vocabulary management system for several years. This was previously permanently integrated in Goobi and could therefore be used in the various areas of Goobi itself, for example within the metadata editor and in some plugins. In the past, the vocabulary server also had an api to allow external applications to communicate with it. However, this has not yet been utilised.

New requirements now presented us with the challenge that functionalities were needed that required more than just an adaptation of the existing vocabulary management. For this reason, we developed a new vocabulary server from scratch that is extremely flexible, recognises different data record types including validation and is designed to be multilingual from the outset.

It also has an export mechanism in RDF format and a migration tool to move the data from the previous vocabulary management system to the new vocabulary server. And, of course, the new vocabulary server also has a very comprehensive REST API that enables full communication with the vocabularies. Goobi workflow now also communicates with the vocabulary server via this API.

One of the side effects of this change is that the vocabulary server can now be operated independently and on its own infrastructure. This means that other applications from different network segments can now actually work with it without Goobi workflow itself having to become publicly accessible.

The Goobi viewer has not yet been connected to the vocabulary server, but this is planned for the coming year. What is also still open is the possibility of defining new vocabularies from the user interface as before. Vocabularies are currently defined in the vocabulary server via REST calls. There will certainly be a user interface for this soon too. Until then, however, we have already provided extensive documentation describing the installation and administration of the vocabulary server. It can be found here:

Like Goobi worklow, the vocabulary server has also been published and documented as open source on Github under GPL2. The repository can be found here:

Conditionally displayed process properties

The process properties are used in many Goobi installations to record information from processors while tasks are being carried out in Goobi. Typical such information includes the opening angle, the use of the glass plate, the device used and some others.

It has often been requested that the fields that can be entered here should not always be listed in full, but should only be displayed to the user when they are required, depending on the selection of another value in another property. We have now implemented this functionality.

For example, a property is displayed within the creation screen:

If the value of this property is changed, other properties are also displayed depending on this:

The same principle also works within the accepted tasks. A property is also displayed there:

And other properties can also be displayed depending on the selection of the property:

The configuration for the example shown here looks as follows within the configuration file goobi_processProperties.xml:

Process title can be hidden in creation mask

Many Goobi users are not aware that I can also hide fields in the creation screen. A new addition this month is that the process title can also be hidden. This can be useful, for example, if it is generated automatically and should not be changed by users.

The configuration is done within the configuration file goobi_projects.xml and can be done there together with the other hiding options:

Redesign of the configuration for the creation screen

We have also made other changes to the Goobi creation screen. The complicated configuration within the goobi_projects.xm file has been a thorn in our side for some time, so we are now allowing new, shortened spellings for the same configuration that are more intuitive to use. Nevertheless, the mechanism remains downwards compatible and can be gradually converted to the more readable configuration.

• If the attribute autogenerated=‘true’ is set for a field (e.g. for the identifier), this field is now hidden by default, but can also be shown by switching the display.

• If the attribute autogenerated=‘true’ is set for a field (e.g. for the identifier) and it is displayed, the label is now followed by a note that it is generated automatically.

• It is now possible to select values from a vocabulary. The attribute ‘vocabulary’ has been introduced for this purpose.

• The previous designations process, work and template within the from attribute have been changed to process, work and template. However, the old attributes will continue to work so that no conversion is necessary.

• The ughbinding attribute is now optional. It is true by default if there is a value in metadata. However, the attribute can still be listed and will be interpreted accordingly.

• The attribute docstruct now has the default value topstruct, so that the attribute can be omitted in most cases and only needs to be listed if firstchild is to be used there.

• A new attribute type has been introduced, which has the default value text. It is only evaluated if no selection list (with or without vocabulary) is used. It allows the values text, textarea, date and datetime.

• If either date or datetime is specified as type, a pattern attribute can also be specified, which defines the layout of the date and time. This is always required for datetime in order to be able to enter the time.

Previous configuration of a main title:

New abbreviated spelling for the main title:

If a property is to be stored directly in the process, the attribute `from can also be omitted:

If, on the other hand, information is only to be saved within the METS file, saving as a property can be prevented in this way:

We have also simplified the notation of the position where a metadata is saved. Previously, the position was expressed in this way:

In a nutshell, it now looks like this:

New field types for the creation screen

Until now, the field types in the Goobi worklow creation screen were quite limited. We have now created three new variants.

Information can now be entered in a text area:

Information can be taken from vocabularies:

And dates with calendar selection are now also possible:

Plugins

Conversion of documentation platform for plugins

Currently, 143 of the 360 plugins already have very detailed bilingual documentation. In order to be able to continue to maintain this well with this large number of usage scenarios and plugin types, we have decided to revise the type of documentation. Instead of maintaining the documentation in a central location, we have moved all documentation to the docs subfolder of the respective plugin and homogenised its structure. As a result, it is now possible for the documentation for each plugin to be available in the same source code repository for the developer in order to have an overview of it and to be able to maintain its content.

We now automatically generate a central data source in a separate repository from this individual documentation, which is therefore much easier to maintain:

Creation of plugin documentation for approx. 20 plugins

And because we have been working so actively on the documentation of plugins anyway, we have simply documented another 20 plugins this month, which means that around 160 plugins are now already extensively documented.

Completion of the new archive management plugins

Over the last few months, we have been working very intensively on revising the archive management plugin. The aim was to implement a number of functional and operational innovations. We have now finally been able to finalise these and put them into operation in a first project.

Some of the biggest innovations in this context include

• No additional BaseX database is required. Instead, Goobi stores the information of all nodes of the archive tree within the Goobi database.

• Several people can work in an archive tree at the same time without having to lock the entire tree.

• EAD2, EAD3 and EAD4 formats can now be read in and processed

• There is now an independent EAD export to the Goobi viewer

• Nodes and tree sections can now be reassigned within the tree

• Nodes can be duplicated

• Nodes can be created in bulk

• Very granular rights can be assigned for working with archive management

• Authorisations can be assigned for content

• Search for all metadata instead of just title content

• Composite fields as groups with multiple metadata fields

• Different field types per metadata item

• Periodic export of the entire EAD inventory to the Goobi viewer

• Highly configurable export for the entire inventory or per metadata

More information about the plugin, its installation and configuration can be found here: https://docs.goobi.io/workflow-plugins/en/administration/goobi-plugin-administration-archive-management

New plugin for enriching MIX metadata

We have implemented a new plugin for enriching METS files with MIX metadata. It allows the metadata to be read from existing images and then saved in the METS file as MIX data. To do this, the plugin first calls JHove and extracts the desired metadata from the analysis result in accordance with the specified configuration.

Here is an excerpt of such a configuration:

In this way, the metadata is transferred to the Goobi METS file. In the further processing of the processes, especially with regard to publication, this configured technical metadata transferred from the master images is then available in the METS file and thus also conveniently available to users.

For example, they can be displayed next to the images or searched for.

Version number

The current version number of Goobi workflow with this release is: 24.08.3. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

September 2023

September

Core

Usability

The Goobi workflow creation screen has always prevented a process with the same name from being created again. You would then receive a short error message saying that the process already exists. This has been enhanced so that you can now also view the details of the existing process.

GoobiScript

The GoobiScript calls metadataAdd and metadataReplace can now also create or manipulate authority data values. This makes it possible, for example, to subsequently adjust the GND ID for persons in bulk.

At the same time, the GoobiScript metadataReplaceAdvanced was introduced. This call also allows you to work with regular expressions.

REST

September also brought changes to the REST interface, two new endpoints were added. The first allows a search and returns a list of process IDs:

The second allows you to complete tasks with a defined title:

Plugins

Import: Excel to EAD

An import plugin has been developed to create an EAD archive structure with metadata from an Excel file.

• GoobiScript templates

• Documentation of step details

Core

API Token

In the settings of one's own user account, there is a new tab in which an API token can be generated and equipped with the corresponding rights.

Only super administrators or user accounts that belong to a group to which the corresponding right has been assigned have the right to create a token.

This token can then be used with a basic auth to query the previously enabled commands.

Create process

When creating a task, the mask can be configured via the file goobi_projects.xml. Multiselect fields could also be defined with the attribute multiselect="true". If several values were then selected, they ended up one after the other in the configured metadatum or the defined property.

As of this release, a separate metadatum or property is written for each value.

Plugins

Step: OCR to metadata

A new step plugin has been developed that allows to convert the OCR text of all images of a process into a metadatum.

Snippets

• ALTO editing in the metadata editor can now also save files in S3.

• The plugin for automatic PDF extraction has received minor improvements, for example in the case of missing spaces in the full text.

• Documentation of step details

• Improved PDF-A generation

Core

User interface

We have worked several hundred hours in the background on it and in the result only little changes at the surface first of all... In the substructure, however, a lot: Where source code was copied and modified over time, there is now a unified system again. New structures have been built, redundancies removed, code and markup for the same things unified and thus the whole structure simplified.

Probably the first thing you'll notice is that everything is a bit more airy and here and there some things might even look a bit strange now. Unusual in any case. But the basic structure and also the design remains the same.

Based on this work, we can now start to work on the usability: How do we want to arrange and name buttons and where? Where do we speak of the terminology in the plural and where singular? Which boxes are displayed where, what is the order of information, and how do we highlight relevant things, hide irrelevant or incidental ones, or reduce the focus? What is a consistent and reliable path to a goal and where do we want to deliberately create shortcuts? With this basis, we can change many things in one central place and it automatically affects all areas. Before we would have had to look at very very many places individually.

Please report things that now look funny or are incomprehensible.

REST API

In April, as part of a project, we began to revisit the REST API and develop what we believe to be essential functionality, refactoring and consolidating existing code as well. Specifically, we considered tasks, steps, properties and the journal and implemented the following functionalities:

• Create tasks, read information, update or delete tasks.

• Read out, update, add, delete or execute steps for an operation

• Read journal and add, update or delete records

• Read out, add, edit and delete properties

The logic uses - where possible - existing code from GoobiScript. The terminology is also chosen to be identical, so there should be recognition effects when working here. The new code has been given meaningful test coverage and the REST commands are also all listed and described in the Swagger documentation.

The new REST endpoints allow integration of functionality with external systems via a standardized interface in the Goobi workflow core.

Here are four example calls for the possibilities available for the journal:

Plugins

For some time now, the plugins installed in an instance can be viewed via the interface. On the page you can also see if the plugin has been tested against the installed Goobi workflow version number.

With the April release, we started to not only test the plugins for compatibility, but also tag them with an identical version number. This way it is now also possible to immediately identify in the plugin repository the commits that ensure compatibility to a specific core version.

We still need to observe the solution we have found now and possibly also readjust it, but we have already come a big step closer to our goal: "To have plugin versions analogous to the Goobi workflow Core".

Code analysis

Version number

The current version number of Goobi workflow with this release is: 23.04.3. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

• Porting of user interface to new framework

• Documentation of step details

• Improved PDF-A generation

• REST API

Core

Periodic tasks

For some time now, Goobi workflow has been able to trigger certain tasks periodically. There are internal tasks that are executed once a day, but also plugins like DataPoller or delay plugins. This functionality works stably and reliably, but had two limitations that we addressed with this release.

First, everything was done in secret. There was no overview of which regular tasks existed, when they ran and if there was an error or not. For this overview, there is now a new page in the "Administration" section that lists the information mentioned. Also an individual execution or pausing of tasks is possible there.

The second part was the configuration options. Either things were not configurable at all or they were given in seconds since midnight. To start something at 15:30 then the value 55800 had to be specified. Not exactly intuitive. Now the cron syntax is used, which is much more intuitive to read and also more powerful in configuration. The syntax for 15:30 every day is now 0 30 15 * * ?

Sorting for additional columns

For some time now, it has been possible to configure additional columns for the table on the "Tasks" and "My tasks" page in the settings of a user account. From now on these are also sortable if they are in the database. This is for example the case for variables like {db_meta.XYZ}, {process.XYZ}, {product.XYZ} or {template.XYZ}. Other variables like {origpath} or even {meta.XYZ} are not sortable.

Usability

The usability issue is progressing in small steps. This month we have taken a look at the project configuration and there especially at the tab "METS Parameters" which has been renamed to "Content settings". The names of the lines as well as the inline help have been massively revised and provided with useful examples.

Snippets

• In the queues, the error table is now automatically and regularly emptied.

• In the project settings, a custom URL for the DFG Viewer METS file can be stored in the "Content settings" area.

• Documentation of step details

• Personal API Token

Core

Error and correction messages

Error and correction messages can be sent in an accepted task. New is that now an optional dropdown menu is available where a list of classic error types can be configured.

The configuration is done for both error and correction messages in the central goobi_config.properties file. Here is an example configuration:

Mail sending

Mail sending from Goobi workflow has been unified and different plugins that can also send emails can now access the same configuration. This eliminates redundancies in the code, minimizes maintenance and makes configuration easier.

Code quality

For some time, we have been working in Goobi workflow with Sonarcloud for static analysis of code quality. In July last year, we started with a good 4,600 issues noted. With this release we have reached about 1,600 issues. In the last month alone, we were able to process around 1,000 issues. In addition, around 75 unit tests were transferred to the core.

Performance

When creating and deleting processes with a lot of steps (let's say more than 50) there were performance problems. These problems could be solved by changing SQL calls and Java code.

Plugins

Import: Excel import

The Excel import plugin now allows the specification of a replacewith attribute for the generation of task titles - analog to the configuration in goobi_projects.xml - in order to replace spaces with underscores, for example.

Step: PDF generation

The PDF generation step plugin now allows specifying and using the PDF generation variants recently introduced in ContentServer. This allows, for example, valid PDF-A files to be generated as a step within a production template and stored in a specified location.

Code analysis

A few words about the failing analysis:

In SonarCloud, quality requirements, called quality gates, can be defined. The quality gates are applied once to the entire source code and once to the source code that has been modified in the last 30 days.

The now failing quality gate in Goobi workflow refers to the adapted source code in the last 30 days. There, a test coverage of at least 25% is required and the duplications should be below 10%. Since we have touched a lot of source code with this release, we do not meet these requirements - based on the source code of the last 30 days.

However, the quality gate for the entire source code is met.

Version number

The current version number of Goobi workflow with this release is: 23.05.1. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Core

Usability

We are always looking for small areas in the user interface in order to specifically improve usability. In August, it was the area of actions within an assumed task, where we tried to create clarity simply by adapting the terminology used.

Here are two screenshots of the area before the update:

There was confusion mainly because the first tab was called "Execute task" and the visualisation suggested a button. However, nothing happened when you clicked on it. The second confusion arose in the area of errors and correction messages, where a "correction requested" and an "error solution reported" were displayed. The expectation here was rather that you would "report an error" and "send a correction". The user interface was made clearer simply by adapting the corresponding translations. Here is the result:

GoobiScript

The GoobiScript openNextStep has been added to set the first locked task of a process from the hit set to open if no other task is open. If it is an automatic task, it is also started automatically.

In addition, the addStep GoobiScript now has the option of using the number: end setting to insert the task at the end of the process, even without knowing the exact number. This is determined automatically during the setting.

REST

A new endpoint has been added to the REST API to start open automatic tasks.

In addition, the endpoint for creating new processes has been extended so that metadata and properties can optionally be added at the same time.

Plugins

Export: VLM

In addition to a password, the VLM export now also supports the option of using the SFTP functionality with an SSKH key.

Export: Heris

A special plugin for the export of five selected images and metadata as a JSON file was developed for the Federal Monuments Office in Austria.

Import: Nuremberg House Books

A special import plugin was developed for the import of the Nuremberg house books and the conversion of an SQL data structure into METS/MODS files.

Step: Image selection

A step plugin for selecting and defining a sequence of images and saving the result for further processing as a process property was finalised.

Code analysis

Version number

The current version number of Goobi workflow with this release is: 23.08. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Core

Bessere Anzeige von Validierungsergebnissen im Metadateneditor

In Goobi workflow, the rule set can be used to control which metadata may occur, for example, how often, where they are permitted and what content is possible in them. In this way, a quality requirement for the metadata to be captured can be well controlled and adhered to. In order for the user to comply well with these requirements, the metadata editor displays validation messages that indicate problems. In the past, however, these messages were only ever displayed globally, not directly on the metadata in question itself and also not on the associated structural element. In other words: As a user, you were a bit perplexed as to which field was not valid and where exactly. You had to do some searching first.

From now on, there are some major changes at this point. Validation messages appear directly at the field in red colour and point out the problems directly with a helpful message.

And also in the tree display of the structural elements in the left-hand pane, the validation problems are now highlighted and now allow the user to find very quickly the metadatum in question for which there is a message.

If you hold the mouse pointer over a structural element with errors, you will immediately see the corresponding message in a pop-up.

By the way, we also took the opportunity to introduce a new key combination for performing the validation. It is: CTRL + SHIFT + V.

Search slot in the task list finally conveniently usable again

After minor optical adjustments the search slot within the task list of Goobi workflow had become too small, it was a bit too cumbersome to use lately. It was simply no longer possible to keep track of which search terms had already been entered when the search query became a little longer.

As of this month, this circumstance is a thing of the past. The search slot is now basically much longer on regular displays.

And even on mobile devices, the usability is such that you can work reasonably with the search:

Correction of URLs for vocabulary records

Goobi workflow has had a vocabulary management system for some time, which can be used, among other things, to make the metadata editor as convenient as possible by means of selection fields with controlled lists. It is also possible that the values are not only available in a selection field but can also be selected from a hit list after using a search mask.

In such a case, not only the selected value is taken over for the subsequent display of the digitised material in the Goobi viewer. Instead, a URL for linking to the corresponding vocabulary record is also entered.

Unfortunately, a small error had crept in here, so that these URLs may not have been correct. We have now corrected this behaviour so that the URLs are now correct. However, please check whether you have already used this functionality. In this case, the URLs you have already entered should be corrected.

First layout changes go live

We had already announced that we would be making some major changes in the background with regard to accessibility and usability. In the case of the administration screen for users, we are now slowly putting these changes into operation. As announced, this will be accompanied by minor visual changes. This should soon have an impact on other areas as well.

This is how it already looks:

GoobiScript now also for changing metadata within metadata groups

For some time there have been various GoobiScripts for changing data within metadata. Unfortunately, until now this was not easily possible for metadata groups. There have been some changes this month so that the existing GoobiScripts can also be used for metadata groups if the parameters are set correctly.

Since the changes are quite extensive, we have also adapted and extended the documentation. As before, it can be found at the following address:

Calls of scripts in the journal better traceable

Within the journal (formerly process log), the individual activities of a workflow can usually be easily traced. Among other things, the calls of scripts were already logged there in the past in order to be able to trace them better in the event of problems. What was missing, however, was that the variables used were also displayed in their replaced form within the script calls. This has now been adapted so that after each call of a script within the journal, both the script call in its original form and in its resolved form are listed.

In the case of a simple script like this one:

... this is what the call within the journal now looks like:

By the way, we also took the opportunity to adjust the height of the box so that it makes better use of the available space and you can see more information at once without having to scroll.

Super admins now always have all rights

Goobi workflow has a fairly granular rights system. It is so flexible that even plugins can define their own rights that a user must have in order to work with the plugin.

To make life a little easier for the administrators (including ourselves), we have now introduced that all users who have the status of 'super administrator' automatically have all rights in Goobi, including the individual rights that plug-ins bring with them.

Plugins

New plugin for flexible file validation

As part of our developments in the delivery of electronic deposit copies, we needed a flexible way for files uploaded by users to be validated and checked for compliance with requirements. Among other things, it should be possible to use any external tools for validation and to ensure compliance with defined levels.

The result of the development is an extremely flexible validation plugin that can be configured very granularly. We have published extensive documentation for the plugin here, including several useful configuration examples:

And as usual, the source code of the plugin and a compiled version can be found on GitHub:

Documentation

URN plugin now allows use behind a proxy

This should only affect a few users, but for those it is important:

The still quite new plugin for URN registration could not be used until now if the Goobi server is only allowed to access the internet via a proxy. Here we have now made an adjustment that from now on also allows operation behind a proxy.

We have updated the documentation for this change:

The plugin can be found as usual here on GitHub:

Documentation for the Artist Dictionary Plugin is online

After a long development on the plugin for capturing linked entities, we have now also completed the related plugins and published them on GitHub. On the one hand, this is a special dashboard plugin:

On the other hand, this workflow plugin is also required:

The installation and configuration of these plugins are now also fully and very granularly documented. They are available as usual on our documentation platform:

Version number

The current version number of Goobi workflow with this release is: 22.10. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>22.10</version>
</dependency>

• Porting of user interface to new framework

• Periodic tasks

• Documentation of step details

Core

Performance

A bottleneck in the performance of SQL queries with "Like" queries could be identified and fixed if. This was the case, for example, when queries for step titles were executed. Loading times could be reduced from 155 seconds to 3.2 seconds in some cases.

A significant speed increase was also achieved in the area of automatic tasks by changing the SQL queries.

The result is a faster responding user interface in certain situations.

Search

Users can now configure individual search entries. For this purpose, a new input area for individual search settings is available in the user settings in the "General" area.

Each line is structured according to the name=search filter principle where the placeholder {} stands for the search term. An example configuration could be as follows:

Default={}
Eigenschaft={} |processproperty:Template:{}
Projekt={} |project:{}
mit Leerzeichen="{}" "|processproperty:Material type:{}" 
Metadaten={} |meta:TitleDocMain:{}

If configured, a drop-down menu with the configured values appears in front of the search slot above the table on the "Operations" page.

Plugins

Step: Rename files

The file renaming plugin used to rename automatically in all task folders. Now it can also work only in certain folders. The use case is that one has different upload steps, the files are stored in each case in a temporary folder, then the plugin names the files individually and these are then moved with a script into the _master folder. This way you can ensure an individual and yet uniform naming of the files.

Step: Change workflow

The plugin for changing the workflow based on task properties can now also influence the priority for individual or all steps of a task. The background to this enhancement is that within a workflow the decision is made whether the task has to be put on the fast lane or not. If overtaking lane, then all steps in the task get a higher priority and appear accordingly higher up in the tasks.

Version number

The current version number of Goobi workflow with this release is: 23.02. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.02</version>
</dependency>

• Porting of user interface to new framework

• Periodic tasks

• Documentation of step details

Core

ALTO Editor

The ALTO Editor for OCR correction, which can be opened in the Metadata Editor if required, now also supports right-to-left languages. Switching is done automatically based on the selected setting in the Metadata Editor itself.

Usability

Minor usability changes are gradually being incorporated into the interface. In this digest we would like to go into a tiny sub-area and show what this means in practice. The starting point in this case is the metadata editor and the possibility to add another metadata. So far this looked like this:

We have looked at this small sub-area and noted the following points:

• The heading "Create new metadata" suggests by the plural that one can "create" here several at the same time. But this is not possible because in the mask only one can be "created" at a time.

• The word "create" is linked by the terminology rather with "create process". The word "add" is more appropriate here, since a new metadatum is to be added to an existing process.

• Also, the word "new" in the context is actually too much. "Add metadata" is also clear and unambiguous. At this point, the ruleset is not edited.

• "Type" is more likely to be associated with something like "Text, MultiSelect, DropDown, Textarea", but not "Additional title". A metadata of type "Additional title" is weird. Better is simply "Name".

• "Save the changes" says that you have changed something. But something is "created" or isn't "Add" the correct terminology here?

The result of the considerations looks as follows:

Of course, there is much more you can ask yourself at this point:

• Why is the box collapsible and expandable?

• From a UX perspective, is it right to have the cancel and add buttons equally weighted visually?

• Why isn't this simply opened in a modal as an overlay?

• ...

But these questions are more complex to answer and we are taking small steps. This example is just to show how we make things better in various places and little by little.

Plugins

Step: Metadata editing

The step plugin to edit metadata directly from the accepted task can now hide empty fields. This was not possible before and led to very long lists in use cases where the plugin was used to quickly check metadata.

Version number

The current version number of Goobi workflow with this release is: 23.01. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.01</version>
</dependency>

• GoobiScript changes

• Documentation of step details

Core

GoobiScript Templates

It often happens that several GoobiScript calls have to be made in succession. Often the calls are repeated because, for example, in a certain project under certain conditions it is necessary to insert a certain step again and again.

There is now a new mechanism to make these tasks more convenient. In the "Administration" section, the new GoobiScript Templates section has been introduced. There, multiple calls can be elegantly concatenated and saved as a template. Each template has a mandatory name and an optional description.

In the list of GoobiScript templates, the self-defined templates are displayed as badges with their own colour. The description is displayed when the mouse is moved over the badge. The same possibilities for dedicated rights management apply to the templates as to the built-in GoobiScript calls.

Properties during process creation

When creating a process, the process properties can now also be managed via the central configuration file goobi_processProperties.xml. For this purpose, the new XML element was added to the configuration file. The attributes are access="" and template="". The values of the access attribute are analogous to the other options already available in the file. The template attribute controls which production template the setting applies to. The XML element is repeatable to be able to configure different production templates. It is possible to define a * in the template attribute so that the setting applies to all production templates.

With this change, it is possible for the first time to visually separate the metadata that is imported from the catalogue from the properties that are to be collected when creating an operation and are therefore completely different in terms of content. Until now, both were displayed mixed together in the upper area with the heading "Import metadata".

Here is an example of a configuration and a screenshot of how it had to be done and looked in goobi_projects.xml:

<item from="prozess" multiselect="false">
    Schrifttyp
    <select label="Antiqua">antiqua</select>
    <select label="Fraktur">fraktur</select>
    <select label="kein OCR">no_ocr</select>
</item>
<item from="prozess" required="true">Fachcode</item>
<item from="prozess" multiselect="false" required="true">
    Digistandort
    <select label="Haus 2">Haus 2</select>
    <select label="Haus 1">Haus 1</select>
</item>
<item from="prozess" multiselect="false" required="true">
    Öffnungswinkel
    <select label="180">180</select>
    <select label="110">110</select>
</item>
<item from="prozess" multiselect="false">
    Glasplatte
    <select label="Ja">true</select>
    <select label="Nein">false</select>
</item>
<item from="prozess" multiselect="false" required="yes">
    Viewerexport
    <select label="Nein">false</select>
    <select label="Ja">true</select>
</item>

The same configuration can now be mapped via goobi_processProperties.xml. A separate box is then displayed for this purpose in which the settings can be made. This makes it possible to realise a more clearly structured and thus clearer interface.

<?xml version="1.0" encoding="UTF-8"?>
<config_processProperties>
    <property name="Schrifttyp" container="0">
        <project>*</project>
        <showProcessCreation access="write" template="*" />
        <type>List</type>
        <value>Antiqua</value>
        <value>Fraktur</value>
        <value>kein OCR</value>
    </property>

    <property name="Fachcode" container="0">
        <project>*</project>
        <showProcessCreation access="writeRequired" template="*" />
        <type>Text</type>
        <defaultvalue></defaultvalue>
    </property>

    <property name="Digistandort" container="0">
        <project>*</project>
        <showProcessCreation access="writeRequired" template="*" />
        <type>List</type>
        <value>Haus 2</value>
        <value>Haus 1</value>
    </property>

    <property name="Öffnungswinkel" container="0">
        <project>*</project>
        <showProcessCreation access="writeRequired" template="*" />
        <type>List</type>
        <value>180</value>
        <value>110</value>
    </property>

    <property name="Glasplatte" container="0">
        <project>*</project>
        <showProcessCreation access="write" template="*" />
        <type>List</type>
        <value>Ja</value>
        <value>Nein</value>
    </property>

    <property name="Viewerexport" container="0">
        <project>*</project>
        <showProcessCreation access="writeRequired" template="*" />
        <type>List</type>
        <value>Nein</value>
        <value>Ja</value>
    </property>
</config_processProperties>

Snippets

• The newspaper export for the DDB now writes extended information to meet DFG viewer display requirements.

Code analysis

Version number

The current version number of Goobi workflow with this release is: 23.07.2. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.07.2</version>
</dependency>

Projects

easydb

Together with a partner we developed a process chain that allows to import records exported from an easydb fully automatically into Goobi workflow, to process them and to export them into a Goobi viewer. For this process chain three plugins were used and extended to meet the requirements of the project.

The process chain also works periodically, creating new operations or updating existing ones. Once imported, records proceed through the workflow steps as usual.

Core

Plugins in the metadata editor

The core of Goobi workflow has been extended to offer plugins within the metadata editor. These plugins are integrated in the blue header at the top and provide their functionality.

The decisive factor for the development was the requirement to be able to change the publication type within the metadata editor. For this, template processes should be created and certain, marked metadata should be taken over during the change.

Metadata generation

It happens again and again that certain metadata are to be formed automatically from other values. An example could be the main title of a letter, which should always be generated as follows: "Letter from [sender] to [recipient] at [place], [date]".

To provide this functionality in the metadata editor the configuration file goobi_metadataDisplayRules.xml was extended by the possibility to generate values from other metadata values for certain metadata under configurable conditions.

A configuration for the above example could look like this:

PDF download in the metadata editor

Within the metadata editor there is now the possibility to generate a PDF of all images at any time. For this purpose there is a new PDF button in the left column with the structure tree

Plugins

Step: Rename process

There is a new step plugin to rename a task within a workflow. For this purpose, the variables available in Goobi workflow can be used. Additionally underscores and minus are allowed as characters. Empty variables are skipped and spaces are removed.

Export: Newspaper export

The plugin for exporting to the DDB newspaper portal has been adapted for better display of data in the DFG Viewer.

Documentation

Version number

The current version number of Goobi workflow with this release is: 22.11. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Announcements

December was marked in the project by the upcoming end of the year: A lot of housekeeping and the attempt to tidy up and finish small things to start with air and a clean desk / project first into the vacations and then also so into the new year. For this reason, there is nothing new to report in the December digest of feature developments. Therefore we want to dare an outlook on what we see for the coming year.

Outlook

User Interface

For a long time we have been working in the background to change the existing user interface. Many hundreds of hours have already gone into it and the result will look more or less the same as before. However, this work will allow us to work on the surface much easier. Changes no longer have to be made and checked in many individual places, but can be implemented in one central place and then affect all places.

As soon as we are done with this work, we want to work on the usability in the interface. This will cover a number of different areas. Examples are the naming of functionality that should be consistent everywhere or the use of always the same icons for the same action. But also the consistent placement of control elements across the application is part of this, keyword: "Expectability".

Finally, visual redesigns are also part of it. More on this later.

REST API

We want to remove the deprecated Command plugins and replace them with the REST API. In addition, the possibilities that are available via the REST API will be massively expanded.

For example, it is planned to give access to functionalities from GoobiScript via REST. This could then, for example, trigger a mass export from scripts or allow the manipulation of metadata in individual processes.

This work should be an impulse to rethink the software architecture as well.

File access

There are many places in the user interface where one can, may, should or would like to access the files of a process. Over the years, various places have been formed here, but they are all rather shortcuts and isolated solutions and a straightforward path is missing. In the coming year, we would like to address this at various levels. We want to collect the places in the user interface where access to the file system is possible and ask ourselves why the functionality is needed at exactly this place. Then we want to design and develop a central way to access the file system and last but not least we want to create a central place in the Java code that will be used for all file operations.

Version number

The current version number of Goobi workflow with this release is: 22.12. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Core

General layout changeover to Bootstrap 5

We already announced it. Some users may have already noticed that the layout will change minimally (at first): We are gradually changing the layout of Goobi workflow to Bootstrap 5. The first pages have already been moved.

But who would have thought it: it is more elaborate than expected. So it will continue to accompany us as a theme for a bit.

Cleaning up the display of active sessions

We have already made some changes to the login and logout area in recent months, including the display of active users and logging. However, we noticed that after logging out, the user's session was not completely cleaned up immediately. We have adjusted a few things here and then put them through their paces. We are now satisfied with this, the storage space for the sessions is now somewhat smaller and, from our point of view, it can stay that way.

Database connection for production templates optimised

On some Goobi systems, especially those with a large database, it was noticeable that the listing of production templates was significantly slower than the listing of tasks. We took a closer look at this this month and were able to determine that the more extensive rights check of the project memberships had a significant influence here. Fortunately, this could be remedied quite well, so that now no performance losses should be noticeable in the production templates.

Password length check corrected

Unfortunately, we noticed a small bug concerning the checking of the password length of users. The length of the password was incorrectly checked - and noted as too short - even if data of an existing user was to be changed without a field for password entry being displayed in the user interface at all. We have corrected the error in this release.

Automatic unit tests

To ensure the quality of software, software developers can implement automatic tests. Such tests can then automatically check individual functionalities of the overall software and are run at least before a release is published.

Goobi workflow has had several hundred such tests for many years. Nevertheless, it is our goal to cover as much of the source code as possible with tests in order to ensure code quality on the one hand, and on the other hand, to always be sure that the existing functionality is still guaranteed in the case of new developments and adaptations.

With this month's release, about 230 new tests have been added. The remarkable thing here, however, is not just the number of tests alone. The newly added tests relate to one of the most complex areas of Goobi workflow, in that we have now also achieved significantly better test coverage in the metadata editor.

Code quality

In addition to increasing test coverage by means of automated tests, we have been working intensively on the code quality of Goobi workflow. To this end, we have now registered Goobi workflow within Sonarcloud, so that we automatically receive code analysis results there and can make successive improvements to the software's source code.

There you can see, for example, how we achieved a level A rating for all important areas of quality analysis this month:

Here you can also see a comparison between before and after:

If you want to take a detailed look at the quality of the Goobi workflow source code, you can take a look here:

Plugins

Extension of the URN plugin for special construction

Not so long ago we reported here about a new plugin for registering URNs that uses the REST API of the German National Library. This plugin has now already gone into regular operation with the first Goobi users. In this context, we have then added an additional functionality: the construction of URNs can now use the timestamp of the registration if desired.

Previous use with a counter:

...
urn:nbn:de:gbv:48-goobi-10
urn:nbn:de:gbv:48-goobi-11
urn:nbn:de:gbv:48-goobi-12
urn:nbn:de:gbv:48-goobi-13
...

New possibility with the use of a timestamp:

...
urn:nbn:de:gbv:48-goobi-2022-08-17-06-30-15
urn:nbn:de:gbv:48-goobi-2022-08-17-06-31-17
urn:nbn:de:gbv:48-goobi-2022-08-17-06-31-23
urn:nbn:de:gbv:48-goobi-2022-08-17-06-31-42
...

We have also taken the opportunity to make the log messages in the operation log a little clearer again, so that they all receive a prefix and are thus clearly distinguishable from other log messages, as in this example:

URN PLUGIN: Running URN generation for process ID 1234
URN PLUGIN: Finished URN generation for process ID 1234

And of course we have also expanded the documentary and added the new possibilities:

Here is also the link to the source code and the compiled version of the plugin on GitHub:

Deleting multiple images in the Image-QA plugin

We have found a malfunction in the popular Image-QA plugin. There, it is possible to delete images, provided that this has been activated. Apparently an error had crept in here, which caused that when selecting several images for deletion, not always the selected images were deleted. This has now been corrected.

More information about the plugin can be found here in the documentation:

And if you don't use the plugin yet, you should definitely try it. You can get it here:

Export-Package plug-in equipped with ZIP file functionality

With the Export-Package plug-in, a plug-in is available that allows almost all conceivable exports of data to third-party systems. It is possible to control exactly which data should be included in the export, where the data should be delivered and much more. Even a conversion of the METS files into third formats can be done by applying an xslt transformation. What was not yet possible, however, was that the exported result should not only be exported as a directory but also zipped afterwards. We have only developed this function and added it to the documentation accordingly.

Here you can find the detailed documentation of the plug-in:

And as usual, the source code of the plugin is here, together with the compiled version of the plugin:

New plugin for mass import from Excel files with EAD enrichment

We have implmeented and published a new plugin for the mass import of Excel files. The special feature here is that several structural elements are generated from each Excel file that is imported, one element per line. In addition, an existing EAD tree can be enriched with further nodes, so that an extensive inventory is created.

The functionality of the plugin can be viewed in detail here in the documentation:

And here is the source code of the plugin:

Version number

The current version number of Goobi workflow with this release is: 22.08. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>22.08</version>
</dependency>

Core

Generation of URLs for the SRU Interface and the IIIF Manifest

In the project settings there are now two additional fields in the tab METS Parameter to record the URL to the SRU interface of the Goobi viewer and the URL of the IIIF Manifest for the complete work.

Example entries could look as follows:

http://viewer.example.org/sru/$(meta.CatalogIDDigital)
https://viewer.example.org/api/v1/records/$(meta.CatalogIDDigital)/manifest/

The result within a METS file created in this way is then accordingly as follows:

<dv:links>
<dv:iiif>https://viewer.example.org/api/v1/records/1234567889/manifest/</dv:iiif>
<dv:sru>http://viewer.example.org/sru/1234567889</dv:sru>
</dv:links>

Adjustments for the image comments

Not too long ago we introduced the new image comments here. With their first use at various institutions, however, we noticed that the internal storage location for the data was not ideal. For this reason, we have made a change for this. Those who have already used this functionality should therefore follow our update instructions provided here for the next update:

GoobiScript metadataChangeType fixed

In the GoobiScript metadataChangeType the parameter any did not work as expected. This has been corrected and now allows metadata to be changed at all levels of a METS file as desired.

For the sake of completeness, here is the link to the detailed documentation of GoobiScript:

Improved logging during application stop

Administrators love logging. It helps to better understand why a problem occurred in case of an error and what can be considered as the trigger for a failure. For reliable operation of productive applications, meaningful logging is not only helpful, but often simply mandatory. It is all the better if the developers of a software work closely together with the administrators who actually run the software.

The result of such cooperation in the context of Goobi is that we are constantly trying to improve the logging of Goobi workflow. This month we took a closer look at the logging of the application stop and made some improvements. After all, sometimes too much logging is not right.

Plugins

Processing newspapers

We had already reported on this at the Goobi Days. In recent months, however, our developments for processing newspapers have been used more and more frequently. That is why we would like to explicitly address this topic again together with the latest developments:

For some time now, it has been possible for Goobi to recognise newspaper editions semi-automatically and to index them with metadata. To this end, the workflow first automatically recognises newspaper editions on the basis of trained models. We have recently expanded this considerably and fed it with more ground truth data. Thus, the recognition accuracy of newspaper editions has improved many times over.

After this automatic recognition, a plugin is available in Goobi worklow for enriching the newspaper editions with edition-related metadata. This makes it relatively easy to record issue numbers, dates and publication days, including morning and evening editions, without having to do this individually for each issue. Instead, a semi-automatic mechanism takes effect here, which counts up the subsequent issues accordingly with a few entries from a user, enriches them with dates and also names them.

In addition, supplements can also be created quickly here, so that a comprehensive METS file is subsequently generated, which has standardised date information and can be searched in the Goobi viewer, e.g. via a calendar entry.

To meet the requirements of the German Digital Library and the DFG Viewer, by the way, an additional export plugin was recently added that prepares and provides the data according to the specifications.

We have worked a lot on the individual components described here in the last few months and have repeatedly optimised their operation. Some things will continue to happen here in the coming weeks. We would be happy to present this in detail in a workshop at the upcoming Goobi Days.

Mass imports with archive stock enrichment

Up to now, we had not even mentioned the various newer import plugins that we had recently implemented for some specific mass imports. For these plugins, there is no detailed stand-alone documentation yet.

However, it is important for us to point out that there are several new mass imports that not only create processes in Goobi workflow, but also increasingly create or enrich entire archive trees with tectonics.

Depending on the project, structured METS files can also be created here, as with other importers for Goobi workflow, where structural elements and associated metadata are imported together with authority data for the associated image files.

Better full text extraction from PDF files

The step plugin intranda_step_pdf-extraction allows full texts to be extracted from PDF files and saved as ALTO files. However, we recently noticed an error in the processing, which was responsible for inappropriate gaps appearing within some words, among other things due to an existing hyphenation in the words.

We were now able to find and fix the cause of this problem. By the way, the plugin was already documented here:

We have published the plugin at the following URL, where the corrected version can now also be downloaded:

Documentaries

A lot has happened again in the area of our documentation:

Inline documentation for GoobiScript

In the GoobiScript addToProcessLog the included explanations were not ideal. There was a small correction here, which now looks like this:

Small change in the update instructions

In order to make the update instructions for Goobi workflow more intuitive, they are now no longer structured by date, but by the respective published version. This change only applies to all future notes that are included there and not retroactively. It should hopefully make it a little clearer for administrators in the future.

Configuration files

We have again put a considerable amount of energy into documenting more configuration files. Almost all configuration files are now documented there in an exemplary manner. Some remaining work is still to be done. But if you want to have a look, here are some sample links to new documentation:

Configuration for dockets:

Configuration of the queues:

Configuration for properties:

Configuration for translations:

Configuration for the Web Api:

Configuration for the authority data:

Configuration for the field display in the metadata editor:

Version number

The current version number of Goobi workflow with this release is: 22.07. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>22.07</version>
</dependency>

Core

Saving in the metadata editor

Within the metadata editor, we noticed that different browsers reacted differently when users clicked on the save button. Specifically, the Chrome and Safari web browsers did not perform any function when the Save button was clicked, while Firefox, for example, worked correctly.

Incidentally, the button with the title 'Save and return' was not affected by this problem..

After some back and forth and a lot of research into the causes, we were finally able to solve the problem and provide a fix with the release.

New journal functionality

For 8 years now, Goobi workflow has offered the possibility of tracking various information and communications via the process log. It has also been possible for some time to upload files (e.g. for restoration documents). This was expanded about 1.5 years ago to include the option of saving damage documents when creating processes.

In other words: the process log is simply practical. So practical that we wanted to have it in other places as well. To do this, we had to foolishly rename it so that it would no longer be used only for processes.

The result is the journal. Functionally, it has remained the same as for the processes. Now, however, it is also available for the institutions and, at least in one special case (delivery of deposit copies), also for users.

We can think of other places in Goobi workflow where the journal could be helpful (e.g. for projects). Perhaps there will be an opportunity to do this soon, because someone else would find it just as practical. In that case, we will be able to expand the journal and report back here.

Accelerated loading behaviour for extensive works

Especially in the case of very extensive works, there were always enormous delays in the loading speed when using S3 as a storage system. The reason for this is that an S3 system has no folder hierarchy and all files instead have enormously long file names. This looks like this, for example:

s3://bucket/123/images/bla_media/00000001.tif
s3://bucket/123/images/bla_media/00000002.tif
s3://bucket/123/images/master_bla_media/00000001.tif
s3://bucket/123/images/master_bla_media/00000002.tif
s3://bucket/123/images/bla_jpg/00000001.tif
s3://bucket/123/images/bla_jpg/00000002.tif

This deviating structure meant that when determining the master directory, for example, all files always had to be run through in order to check whether the directory name being searched for was represented. Especially with very extensive works, performance suffered greatly.

We have now changed this so that when searching for folders in the desired hierarchy level, everything after the next slash ( /) is no longer taken into account and so when searching within the images folder, significantly fewer file names to be checked are listed.

In addition, we have now always limited the file search to the actually relevant folder, so that even when searching for the master images, the data of all folders of a process are no longer queried.

Another big improvement in performance resulted from the fact that in the IIIF API we now no longer check per image whether a directory with the name 'thumbs' exists, but cache the result for a while. This has also allowed us to significantly reduce the number of requests to the system once again.

Blemish with empty table columns removed

During the detailed display of workflow information of individual work steps, it occasionally happened that empty table columns were displayed.

We have checked this once and revised it. This means that this small display error no longer occurs.

Better check of the available storage space

We noticed that the metadata editor did not display a user-friendly message if there was not enough storage available on the Goobi server to save the metadata while editing the metadata. Ideally, of course, this should not happen at all in everyday life. However, if it does happen, Goobi now notices the lack of storage space better and displays an easily understandable message for the user.

Interaction with a proxy improved

If it is necessary for Goobi itself to communicate with other services via a proxy, this was already possible quite easily in the past. However, a new requirement has arisen due to the increasing use of archive management. This requires communication with another local database, which in turn does not require communication via a proxy.

With this version, therefore, a new configuration option has been added, which from now on allows Goobi not to use the proxy for communication with its own server (localhost). This is configured with the new switch as follows:

http_proxyIgnoreHost=localhost

Extended LIDO support

If Goobi is used to use the LIDO format instead of METS, especially in the case of museum objects, from this version onwards it is possible to write empty elements within the generated XML files. For this purpose, a new configuration option has been created for the rule sets, internally specifying the following there:

<exportEmptyNode>true</exportEmptyNode> 

Illustrated by a concrete example, this looks like this:

<Metadata>
    <InternalName>OtherTitle</InternalName>
    <exportEmptyNode>true</exportEmptyNode>
    <XPath>./lido:objectIdentificationWrap/lido:titleWrap/lido:titleSet/lido:appellationValue[@lido:pref='alternative']</XPath>
    <WriteXPath>./lido:objectIdentificationWrap/lido:titleWrap/lido:titleSet/#lido:appellationValue[@lido:pref='alternative']</WriteXPath>
</Metadata>

The result within the generated LIDO would then look like this:

<lido:objectIdentificationWrap>
    <lido:titleWrap>
        <lido:titleSet>
            <lido:appellationValue lido:pref="preferred">Testdatensatz_LIDO, Zusatztitel</lido:appellationValue>
            <lido:appellationValue lido:pref="alternative" />
        </lido:titleSet>
    </lido:titleWrap>
</lido:objectIdentificationWrap>

The configuration made here has the following influence:

If the metadatum in question exists, it is exported normally. If, on the other hand, it does not exist, it is checked whether it is permitted in the current structure element. If this is the case, 'WriteXPath' is also executed. Such a configuration can also be specified for a group or for a metadatum in a group.

If such a configuration is not made, this line is omitted in the generated LIDO:

<lido:appellationValue lido:pref="alternative" />

More on the configuration options for the metadata exports can be found in the following documentation:

Plugins

Plugin for editing personal data nears completion

Again this month, a lot of energy has gone into the completion of a new plugin for the granular collection of personal data. In addition to various minor and major bug fixes, the user interface, among other things, was once again thoroughly checked and adapted.

We expect to complete the plugin in October and to be able to document it thoroughly then. We will keep you up to date.

Extension of the Catalogue Poller

The Catalogue Poller is used to automatically update the metadata within the METS files periodically for numerous processes. In this way, catalogue changes can be automatically adopted and are subsequently also taken into account for the presentation of the digital copies. This plugin has now been fundamentally revised and equipped with further functions. In addition to some changes to the user interface, the main addition here was the new possibility that the Catalogue Poller can now also carry out test runs, which make it possible to see what changes would be made to the data without actually changing it. The result of such a test run can be downloaded as an Excel file.

Also new is the possibility that within the configuration of the metadata to be taken into account, it is no longer possible to name only those fields that are not to be changed (blacklist). Instead, it is now also possible to name only those fields that are to be changed (whitelist). Here is an excerpt from the configuration:

      <!-- fieldList: Must have a mode attribute which can contain either blacklist or whitelist as a value.
        blacklist: All fields are updated except the defined ones. This is a potential dangerous setting!
        whitelist: Only the definied fields are updated. All others are skipped. 
        field: Use the internal metadata names from the ruleset as field definition
      -->
      <fieldList mode="blacklist">
         <field>viewerinstance</field>
         <field>singleDigCollection</field>
         <field>pathimagefiles</field>
         <field>_urn</field>
         <field>_representative</field>
      </fieldList>

The documentation of the plugin can be found in an updated form at the following address:

The plugin can be downloaded as usual from GitHub here:

Plugin for the generation of PDFs extended

For some time now, there has been a plugin that can automatically generate PDFs during workflow execution. Until now, however, its configurability was limited. This has improved with the new version, so that an individual configuration can be defined for each project and title of a work step. This can look like this, for example:

<?xml version="1.0" encoding="UTF-8"?>
<config_plugin>
    <!-- order of configuration is: 
         1.) project name and step name matches 
         2.) step name matches and project is * 
         3.) project name matches and step name is * 
         4.) project name and step name are * 
    -->
    <config>
        <!-- which projects to use for (can be more then one, otherwise use *) -->
        <project>*</project>
        <!-- which stepss to use for (can be more then one, otherwise use *) -->
        <step>*</step>
         <!-- Choose the source images folder that shall be used for PDF generation. Possible values are 'media' and 'master' -->
        <imagesFolder>media</imagesFolder>
        <!-- If enabled, PDF files for every single image are generated -->
        <pagePdf enabled="true" /> 
        <!-- If enabled, a PDF file for the entire record is generated. This file includes the table of contents from the METS file if exists. -->
        <fullPdf enabled="true" />
    </config>
    
    <config>
        <project>testocr</project>
        <step>testpdf</step>
        <imagesFolder>master</imagesFolder>
        <pagePdf enabled="true" />
        <fullPdf enabled="false" />
    </config>
</config_plugin>

In this way, the plugin can also be operated several times in a Goobi in different projects and workflows and with different settings in each case.

New REST plugin for MARC data reference

In the context of our developments for electronic deposit copies, there was a requirement that it should be possible to extract data from selected processes in Goobi workflow from external systems. The following functionality is available for this purpose:

Within the workflow, there is first a step for MARC export, which exports the metadata as marcxml. The following plugin is used for this:

The exported data then ends up in a directory that is monitored by the new REST plugin. This is the following plugin:

Within this REST plugin, various commands are available to enable communication between Goobi workflow and a library catalogue.

The following commands are available:

List all importable data

This command lists all records that are currently available.

Calling the REST API:

curl -s -H "Content-Type: application/xml" -H "token:SECRET" http://127.0.0.1:8080/goobi/api/delivery/listfiles

Response of the REST API:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<files>
    <file>a643b394-d56c-415c-9da9-b07008187445.xml</file>
</files>

Query a specific data set

This command can be used to query a specific data set.

Calling the REST API:

curl -s -H "Content-Type: application/xml" -H "token:SECRET" http://127.0.0.1:8080/goobi/api/delivery/get/a643b394-d56c-415c-9da9-b07008187445.xml

The REST API then responds with the queried MARC data set.

Successful data transfer

This call confirms the successful transfer of the data set and reports the catalogue ID (aDIS ID). The step is then closed.

Calling the REST API:

curl -s -X POST -H "Content-Type: application/xml" -H "token:SECRET" http://127.0.0.1:8080/goobi/api/delivery/finish/a643b394-d56c-415c-9da9-b07008187445.xml/aDIS-ID12345

Reporting a Data Transfer Error

This command is used to report a data set to Goobi whose data transfer was not successful. The error is reported and the work step is set to error status.

Calling the REST API:

curl -s -X POST -H "Content-Type: application/xml" -H "token:SECRET" -d'<msg><type>error</type><message>Corrupt filename</message></msg>' http://127.0.0.1:8080/goobi/api/delivery/error/a643b394-d56c-415c-9da9-b07008187445.xml

With the last two calls, a workflow step is closed or set to error status. The MARC data set is then removed from the export directory so that it is not listed again in the next search.

Documentation

Update instructions for the XMP plugin added

We noticed that the update instructions in July 2020 were still missing a note for the necessary adjustments to the configuration of the XMP header plugin. We have now added these so that everything should be up to date.

As before, the update instructions can be found at the following address:

Version number

The current version number of Goobi workflow with this release is: 22.09. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>22.09</version>
</dependency>

Display of the number of images in tasks and activities

Within the user interface, it is now possible to display the number of images as a column. This can be done within the task list as well as in the own tasks. This looks like this, for example:

Installed plugins better manageable

We have slightly revised the incredibly practical display of installed plug-ins and their versions. All existing plugins are now listed correctly sorted and the paths to the directories are also listed.

Better sortability of tasks and assignments

In the list of tasks, we have made some improvements regarding sorting. The biggest noticeable difference will probably only be noticed by administrators:

All columns now sort lexicographically in ascending order when you activate the sorting of a column. Only the column with the ID behaves differently and sorts in descending order at the first click, in order to display the most recent transactions immediately at the top. Administrators will be pleased. :)

Project structure optimised

Goobi workflow has been using Maven as a build system for a number of years. Last month, the project structure within Maven was further consolidated so that there is a dedicated module for running automated tests and checks within the Continuous Integration infrastructure.

The previously existing modules to build a .war file for deploying the known application within a Tomcat and the module to build a .jar file have been cleaned up and are now only responsible for compiling the artefacts. A new addition is a CI module that bundles previously scattered steps in the lifecycle and also integrates new checks. These are in detail:

• Running the JUnit tests

• Checking the dependencies used for known security vulnerabilities (OWASP)

• Creating an analysis of how much code is covered by unit tests (Jacoco)

• Checking the source code for violations of selected checkstyle rules

• Performing a static code analysis using Sonar

At the same time, the execution of the unit tests was changed from a suite-based infrastructure to a class-based one. The number of regularly checked tests could be massively increased. In addition, the last code sections that required compiling with Java 8 and explicit compiler options were cleaned up.

The changed project infrastructure now allows for faster building of the individual modules and thus less overhead in development, as well as expanded and improved testing of daily work to increase code quality.

Anyone who would like to track our progress in terms of code quality is welcome to do so here at any time:

The diagrams still show that we have a long way to go. But we are on the way.

And internally, progress looks something like this at the moment:

So things are moving forward. We'll report on it here more often in the future.

Better log information about executed GoobiScript

In the past we often noticed that we would have liked to have more logging if a GoobiScript did not do what we had hoped. We have now addressed this issue and solved it in such a way that administrators can easily see in the log files what was called. It now looks something like this:

Extensions for the AEON plugin

Some time ago we had already reported that we were implementing an extensive interface to the software AEON. There have been numerous extensions here again. Among other things, the look of the interface was massively revised.

Properties can be individually adjusted or adopted for entire datasets or even for individual objects that are actually to be digitised, with deviations from the default values clearly visualised. In addition, it is now possible for processes whose digitisation workflow is already in progress to be stopped, unless a configurable status has already been reached.

So, all in all, great progress, but also further work in progress.

GoobiScripts for metadata manipulation extended

There are already a lot of GoobiScripts. But every now and then we notice that a certain functionality would be handy. This time we noticed that it would be quite practical if we could manipulate metadata not only in the logical areas of the METS files, but also within the metadata for the images. No sooner said than done. Besides the parameters work top child and any it is from now on possible for the GoobiScripts to understand and apply physical for manipulating the metadata. More details can be found here in the documentation:

Changes to the database driver

To make Goobi workflow better able to handle newer MySQL databases, we have replaced the driver for communicating with the database. From now on we use the following driver:

It is now possible to run Goobi workflow with MySQL versions 5.7 and 8. However, the earlier versions are no longer supported. This must be taken into account in future updates.

This is already mentioned in the update instructions:

Extension of the Goobi manual

As you can see from our digests and plugins, the documentation of our developments has been very important to us for several years. Currently, we have also turned our attention to the configuration files. This area is still a work-in-progress. But if you want to take a look, you will find a few good examples here:

There will be a lot more happening here in the next few weeks.

Better display of information in the process log

In the case of failed script calls, the process log was not really very helpful until now. This has been adjusted so that incorrect calls now appear with a clear error message in the process log.

New user: Password was not a marked mandatory field

Incorrectly, when creating new users, it was not displayed that a password for a new account was a mandatory field. This has been corrected.

Version number

The current version number of Goobi workflow with this release is: 22.05. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file: