At this point we provide a detailed insight into our current developments. We try to keep them relatively up to date and to summarize for each month if possible what innovations or changes have been made.

In the table of contents on the left you will find a list of the respective months for which such a compilation already exists. And here you find a direct link to the latest digests:

General

Core

Maven structure

We implemented the Maven infrastructure change in April. Goobi workflow still looks the same, but a lot has changed under the bonnet. This is particularly interesting for developers and system administrators.

The new structure now makes it easier to refactor and maintain dependencies in a central location.

Developers

For developers, there is now no longer the goobi-workflow repository alone, but workflow-base and goobi-workflow-core. We recommend that developers delete all local repositories and check them out again from Github. The core and the plugins now define workflow-base as parent. The location of the artefacts in Nexus has also changed. They are no longer located under de.intranda.goobi.workflow, but under io.goobi.workflow.

System administrators

For system administrators, the naming of artefacts in the plugins has changed. The spelling of .jar plugin files and .xml configuration files has been harmonised and standardised. Care must be taken during updates to ensure that no old artefacts are left behind.

Snippets

• A larger set of special plugins was finalised for a customer

• Work in the context of vocabularies and archive is progressing

General

Core

User settings

For some time now, the number of images within a process can be displayed as an additional table column. Until now, however, it was not possible to activate this setting in general, but it had to be reset after each login. From the January release, it will be possible to save the option in the user settings so that it is active by default for an account.

Search

It is now possible to search for all tasks that have a task with a specific title. Previously, the status of the task had to be specified (steplocked:, stepopen:, stepinwork:, stepdone:, ...). From now on, simply step: followed by a title or part of a title also works.

Harvester

The harvester introduced in October has been extended to include a test mode for the OAI-PMH type. If this is activated, only the first page is queried rather than the complete hit set. The resumptionToken is ignored.

Static from and until values can also be set. If the values are empty, the system searches incrementally for all new data since the last query.

Accessibility

In the January release, we worked on many small construction sites in the context of improved usability and accessibility. For example, some page titles have been added, operation via the keyboard under Firefox has been improved, alternative texts for graphic elements have been added and the page structure has been improved. The changes are only noticeable when Goobi workflow is operated with a screen reader or without a mouse.

Security

For some time now, the journal has also been displayed for user accounts. From the January release, a message is automatically left there when the user profile is entered by an administrator.

Snippets

• The steptitle option is now optional in the GoobiScript setPriority function.

• An error that occurred during an empty search for authority data is now intercepted.

Plugins

Import: Excel

The generic import plugin for Excel files can now also process the creation of processes via GoobiScript.

OPAC: Soutron

The Soutron OPAC plugin now also supports the new XML API.

Code analysis

The following screenshot shows the SonarCloud analysis of the current release. More information is available directly on the project page.

Version number

The current version number of Goobi workflow with this release is: 24.01.2. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>24.01.2</version>
</dependency>

General

Core

Config Editor

The newly integrated configuration editor has been extended to allow complete files to be uploaded and downloaded. Furthermore, the backups of the files written in the background are optionally offered for download in the user interface

Periodic tasks

There have been minor changes to the user interface in the Quartz job management section. When a task is paused, the schedule and date for the next execution are now not displayed. In addition, the icon for one-time execution has been changed.

Vocabularies

In the area of vocabularies, work has begun on standardising the usability of importing and exporting data records.

Archives

In the area of archives, work has begun on implementing the new Java logic for searching in collections.

Plugins

Export: ADM - BSME

An export plugin has been developed that supports the creation of special export formats in the Imagen Media Archive Management software.

Step: ALMA API

The ALMA API plugin can now also contain conditions for variables.

Step: Metadata enrichment via Excel file

The new step plugin allows metadata to be read from an Excel file and added to existing structural elements.

Core

Tables

Finally, the column width can be customised within Goobi workflow. The information is saved in the browser's local storage. Once a setting has been made, it is therefore linked to the PC and the browser and is not stored on the server side. This means that different settings can be made on different PCs with different screens so that they do not always get in each other's way.

GoobiScript

If there are several export steps in a process, you can now pass the title in the GoobiScript so that the correct export is triggered. Example:

---
# This GoobiScript allows to export Goobi processes using the default export mechanism. It either uses the default export or alternativly an export plugin that was configured in one of the workflow steps.
action: export

# Decide if the images shall get exported additionally to the metdata (`true`).
exportImages: false

# Decide if the OCR results shall get exported additionally as well (`true`).
exportOcr: false

# Decide which export plugin shall be used by giving the name of this export plugin. It has the highest priority if it is configured. [OPTIONAL]
pluginName: 

# Decide which export plugin shall be used by giving the name of the export step. If it is configured then it will try to find this export step first, and only when there is no such step, will the standard choice be used instead. [OPTIONAL]
stepName: "Export into the Goobi viewer"

Snippets

• Various minor styling adjustments were made in the area of breadcrumbs or menus Initial internal preparations have been made for Java 17 support

• The message queue has received performance improvements

• A discontinued Java library for processing regular expressions that has not been maintained for many years has been removed and replaced by an alternative solution.

Plugins

Step: Downloading and verifying files

A step plugin has been developed to download files, verify them and then provide feedback to the external system.

Version number

The current version number of Goobi workflow with this release is: 23.12.3. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.12.3</version>
</dependency>

Announcements

December was marked in the project by the upcoming end of the year: A lot of housekeeping and the attempt to tidy up and finish small things to start with air and a clean desk / project first into the vacations and then also so into the new year. For this reason, there is nothing new to report in the December digest of feature developments. Therefore we want to dare an outlook on what we see for the coming year.

Outlook

User Interface

For a long time we have been working in the background to change the existing user interface. Many hundreds of hours have already gone into it and the result will look more or less the same as before. However, this work will allow us to work on the surface much easier. Changes no longer have to be made and checked in many individual places, but can be implemented in one central place and then affect all places.

As soon as we are done with this work, we want to work on the usability in the interface. This will cover a number of different areas. Examples are the naming of functionality that should be consistent everywhere or the use of always the same icons for the same action. But also the consistent placement of control elements across the application is part of this, keyword: "Expectability".

Finally, visual redesigns are also part of it. More on this later.

REST API

We want to remove the deprecated Command plugins and replace them with the REST API. In addition, the possibilities that are available via the REST API will be massively expanded.

For example, it is planned to give access to functionalities from GoobiScript via REST. This could then, for example, trigger a mass export from scripts or allow the manipulation of metadata in individual processes.

This work should be an impulse to rethink the software architecture as well.

File access

There are many places in the user interface where one can, may, should or would like to access the files of a process. Over the years, various places have been formed here, but they are all rather shortcuts and isolated solutions and a straightforward path is missing. In the coming year, we would like to address this at various levels. We want to collect the places in the user interface where access to the file system is possible and ask ourselves why the functionality is needed at exactly this place. Then we want to design and develop a central way to access the file system and last but not least we want to create a central place in the Java code that will be used for all file operations.

Version number

The current version number of Goobi workflow with this release is: 22.12. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>22.12</version>
</dependency>

Core

Harvester

Much of the functionality of the intranda harvester has been integrated into the Goobi workflow core. Repositories of different types can now be created and configured. The harvester queries a repository, for example of type OAI-PMH, at a specified frequency and downloads the data records to a specified folder. Repositories can be switched on or off individually.

It is possible to define a script that is to be applied to each downloaded data record. This makes it very easy to modify the downloaded files.

If desired, processes can also be created in Goobi workflow. To do this, a project, a production template and the import format must be selected. The mapping from the rule set is available for the import formats.

This integration allows data records to be harvested from Goobi workflow, for example, prepared via a script and then forwarded directly to the Goobi viewer hotfolder. Another scenario is that, for example, MARC-XML data records are downloaded from an OAI-PMH interface and processes are created in Goobi workflow for further work.

LIDO

The Java library UGH has been extended so that it can also generate empty elements, as is necessary for the LIDO format. With the following export mapping, for example, an element is created even if there is no value:

        <Metadata>
          <InternalName>Person</InternalName>
          <WriteXPath>./lido:eventActor/lido:displayActorInRole</WriteXPath>
          <exportEmptyNode>false</exportEmptyNode>
        </Metadata>

Plugins

Dashboard: Barcode

The dashboard plugin allows you to scan a barcode into an input box and directly execute a previously selected action. For example, a task can be accepted or accepted and directly completed again.

Import: Liechtensteiner Volksblatt

A special import plugin has been developed for importing the Liechtensteiner Volksblatt.

Code analysis

The following screenshot shows the SonarCloud analysis of the current release. More information is available directly on the project page.

Version number

The current version number of Goobi workflow with this release is: 23.10.2. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.10.2</version>
</dependency>

Thumbnails folder per process

Goobi workflow has now been extended so that thumbnails can now also be used in addition to the directories already used for the master files and derivatives. After completion of the entire implementation, these are then used within the metadata editor as well as within the Image QA plug-in for image display and thumbnail display. If such thumbnails are available, the image display can be done much faster. The directory convention for the thumbnails can be shown as follows for an exemplary process testitem_goobi_123:

/opt/digiverso/goobi/metadata/456/
├── meta.xml
├── images/
│   ├── master_testitem_goobi_123_media/
│   │   ├── 000000001.tif
│   │   ├── 000000002.tif
│   │   ├── 000000003.tif
│   │   └── 000000004.tif
│   └── testitem_goobi_123_media/
│       ├── 000000001.tif
│       ├── 000000002.tif
│       ├── 000000003.tif
│       └── 000000004.tif
└── thumbs/
    ├── master_testitem_goobi_123_media_400/
    │   ├── 000000001.jpg
    │   ├── 000000002.jpg
    │   ├── 000000003.jpg
    │   └── 000000004.jpg
    └── master_testitem_goobi_123_media_800/
    │   ├── 000000001.jpg
    │   ├── 000000002.jpg
    │   ├── 000000003.jpg
    │   └── 000000004.jpg
    ├── testitem_goobi_123_media_400/
    │   ├── 000000001.jpg
    │   ├── 000000002.jpg
    │   ├── 000000003.jpg
    │   └── 000000004.jpg
    └── testitem_goobi_123_media_800/
        ├── 000000001.jpg
        ├── 000000002.jpg
        ├── 000000003.jpg
        └── 000000004.jpg

At the same time, it is even easier to clean up the Goobi workflow system. Master files and derivatives can now be deleted from the system to save space. At the same time, however, very strongly compressed files can still be used for the workflow-internal image display. In addition, this also ensures that the thumbnails are not accidentally used for publication to the Goobi viewer and overwrite the high-resolution images there. Such a workflow could be structured in a similar way:

New queue processing in Goobi workflow

Goobi workflow has been enhanced with an internal message queue, which can be used to process complex tasks. With this new mechanism, it is possible to configure how many jobs can run simultaneously, thus preventing unwanted load peaks on the Goobi server. At the same time, it is also possible to use an external queue. This makes it possible to execute orders independent of Goobi on other computers and to distribute computing load across several systems. Previous developments (commits) have so far only introduced this new queue. Shell calls and all Goobi scripts will have to be converted to this new mechanism in the future.

https://github.com/intranda/goobi/commit/af99eebcc7cb0231a528e0434fda66a3da2797b0 https://github.com/intranda/goobi/commit/bf77ef0656eeec4a4a261aa29a5b8ac98455d3b9 https://github.com/intranda/goobi/commit/c051dd18b22163ec7a9b336bd98d5a059519d064

Trigger Goobiscript with Enter

The security dialog for the execution of GoobiScript was already revised not so long ago. However, what still disturbed the operation was that after entering the sum to be calculated, one still had to click with the mouse on the corresponding button to start the GoobiScript.

This has now been revised so that pressing the Enter key already checks the calculation and the GoobiScript starts.

https://github.com/intranda/goobi/commit/27c2764aa5c904cdfae7b16fecde7c5ec5e1f318

GoobiScript to export the extended process log for Goobi-to-Goobi Export

The first important step in the development of data exchange between different Goobi instances has now been taken as part of a more comprehensive import-export implementation. For this purpose, a new GoobiScript exportDatabaseInformation was developed, which writes a very detailed process log to the process directory for each filtered process.

For example, such a log file looks like the following:

<?xml version="1.0" encoding="UTF-8"?>
<process xmlns="http://www.goobi.io/logfile" id="10" template="false" displayInProcessCreation="false">
  <id>10</id>
  <title>alphrudeh_591416573</title>
  <creationDate>2017-06-21T14:36:13.019+0200</creationDate>
  <ruleset id="1" name="Sample ruleset" filename="ruleset.xml" />
  <sorting status="062012025" images="26" articles="0" docstructs="7" metadata="22" mediaFolderExists="false" />
  <docket id="1" name="Sample docket" file="docket.xsl" />
  <project archived="false">
    <id>2</id>
    <title>Manuscript_Project</title>
    <fileFormatInternal>Mets</fileFormatInternal>
    <fileFormatDmsExport>Mets</fileFormatDmsExport>
    <startDate>2017-01-02T00:00:00.000+0100</startDate>
    <endDate>2018-03-31T00:00:00.000+0200</endDate>
    <pages>200000</pages>
    <volumes>150</volumes>
    <exportConfiguration useDmsImport="true" dmsImportCreateProcessFolder="false">
      <dmsImportTimeOut>0</dmsImportTimeOut>
      <dmsImportRootPath>{goobiFolder}../viewer/hotfolder/</dmsImportRootPath>
      <dmsImportImagesPath>{goobiFolder}../viewer/hotfolder/</dmsImportImagesPath>
      <dmsImportSuccessPath />
      <dmsImportErrorPath />
    </exportConfiguration>
    <metsConfiguration>
      <metsRightsOwner>intranda GmbH</metsRightsOwner>
      <metsRightsOwnerLogo />
      <metsRightsOwnerSite>http://goobi.intanda.com</metsRightsOwnerSite>
      <metsRightsOwnerMail />
      <metsDigiprovReference>http://viewer-demo01.intranda.com/Portal/Default/en-GB/RecordView/Index/$(meta.CatalogIDDigital)</metsDigiprovReference>
      <metsDigiprovPresentation>http://viewer-demo01.intranda.com/viewer/ppnresolver?id=$(meta.CatalogIDDigital)</metsDigiprovPresentation>
      <metsDigiprovReferenceAnchor>http://viewer-demo01.intranda.com/Portal/Default/en-GB/RecordView/Index/$(meta.CatalogIDDigital)</metsDigiprovReferenceAnchor>
      <metsDigiprovPresentationAnchor>http://viewer-demo01.intranda.com/viewer/ppnresolver?id=$(meta.topstruct.CatalogIDDigital)</metsDigiprovPresentationAnchor>
      <metsPointerPath>http://viewer-demo01.intranda.com/viewer/metsresolver?id=$(meta.CatalogIDDigital)</metsPointerPath>
      <metsPointerPathAnchor>http://viewer-demo01.intranda.com/viewer/metsresolver?id=$(meta.topstruct.CatalogIDDigital)</metsPointerPathAnchor>
      <metsPurl />
      <metsContentIDs />
      <metsRightsSponsor />
      <metsRightsSponsorLogo />
      <metsRightsSponsorSiteURL>info@intranda.com</metsRightsSponsorSiteURL>
      <metsRightsLicense />
    </metsConfiguration>
    <fileGroups>
      <projectFileGroup id="2" folder="" mimetype="image/jpg" name="PRESENTATION" path="file://{goobiFolder}../viewer/media/$(meta.CatalogIDDigital)/" suffix="jpg" />
    </fileGroups>
  </project>
  <log>
    <entry id="72">
      <content>Step 'Get manuscript from book depot' closed.</content>
      <creationDate>2017-06-21T14:40:59.451+0200</creationDate>
      <type>debug</type>
      <user>Fuller, Cedric</user>
    </entry>
    <entry id="73">
      <content>Step 'Scanning' opened.</content>
      <creationDate>2017-06-21T14:40:59.452+0200</creationDate>
      <type>debug</type>
      <user>Fuller, Cedric</user>
    </entry>
    <entry id="90">
      <content>Script '/opt/digiverso/goobi/scripts/script_createSymLink.sh /opt/digiverso/goobi/metadata/10/images /opt/digiverso/goobi/users/testscanning/alphrudeh_591416573__[10] testscanning' was executed with result: /opt/digiverso/goobi/metadata/10/images
/opt/digiverso/goobi/users/testscanning/alphrudeh_591416573__[10]</content>
      <creationDate>2017-06-21T14:44:25.163+0200</creationDate>
      <type>debug</type>
      <user>Hawking, Delbert</user>
    </entry>
    <entry id="92">
      <content>Step 'Scanning' closed.</content>
      <creationDate>2017-06-21T14:44:40.101+0200</creationDate>
      <type>debug</type>
      <user>Hawking, Delbert</user>
    </entry>
    <entry id="93">
      <content>Step 'Quality control' opened.</content>
      <creationDate>2017-06-21T14:44:40.102+0200</creationDate>
      <type>debug</type>
      <user>Hawking, Delbert</user>
    </entry>
    <entry id="130">
      <content>Script '/opt/digiverso/goobi/scripts/script_createSymLink.sh /opt/digiverso/goobi/metadata/10/images /opt/digiverso/goobi/users/testqc/alphrudeh_591416573__[10] root' was executed with result: /opt/digiverso/goobi/metadata/10/images
/opt/digiverso/goobi/users/testqc/alphrudeh_591416573__[10]</content>
      <creationDate>2017-06-21T14:48:16.539+0200</creationDate>
      <type>debug</type>
      <user>Readdie, Moss</user>
    </entry>
    <entry id="132">
      <content>Step 'Quality control' closed.</content>
      <creationDate>2017-06-21T14:48:22.830+0200</creationDate>
      <type>debug</type>
      <user>Readdie, Moss</user>
    </entry>
    <entry id="133">
      <content>Step 'Image processing' opened.</content>
      <creationDate>2017-06-21T14:48:22.831+0200</creationDate>
      <type>debug</type>
      <user>Readdie, Moss</user>
    </entry>
    <entry id="134">
      <content>Step 'Image processing' started to work automatically.</content>
      <creationDate>2017-06-21T14:48:22.833+0200</creationDate>
      <type>debug</type>
      <user>Readdie, Moss</user>
    </entry>
    <entry id="136">
      <content>Step 'Image processing' closed.</content>
      <creationDate>2017-06-21T14:48:23.412+0200</creationDate>
      <type>debug</type>
      <user>- automatic -</user>
    </entry>
    <entry id="137">
      <content>Step 'Metadata enrichment' opened.</content>
      <creationDate>2017-06-21T14:48:23.413+0200</creationDate>
      <type>debug</type>
      <user>- automatic -</user>
    </entry>
  </log>
  <properties>
    <property id="77" container="0">
      <creationDate>2017-06-21T14:36:34.091+0200</creationDate>
      <name>Creator of digital edition</name>
      <value>Goobi</value>
    </property>
    <property id="79" container="0">
      <creationDate>2017-06-21T14:36:34.091+0200</creationDate>
      <name>Digital origin</name>
      <value>reformatted digital</value>
    </property>
    <property id="75" container="0">
      <creationDate>2017-06-21T14:36:34.091+0200</creationDate>
      <name>Digitisation date</name>
      <value>2017</value>
    </property>
    <property id="76" container="0">
      <creationDate>2017-06-21T14:36:34.091+0200</creationDate>
      <name>Electronic ed.</name>
      <value>[Electronic ed.]</value>
    </property>
    <property id="80" container="0">
      <creationDate>2017-06-21T14:36:34.091+0200</creationDate>
      <name>OCR-Language</name>
      <value>template_without_language.xml</value>
    </property>
    <property id="78" container="0">
      <creationDate>2017-06-21T14:36:34.091+0200</creationDate>
      <name>Place of digital edition</name>
      <value>Göttingen</value>
    </property>
    <property id="82" container="0">
      <creationDate>2017-06-21T14:36:34.091+0200</creationDate>
      <name>Template</name>
      <value>Manuscript_Workflow</value>
    </property>
    <property id="83" container="0">
      <creationDate>2017-06-21T14:36:34.091+0200</creationDate>
      <name>TemplateID</name>
      <value>1</value>
    </property>
    <property id="81" container="0">
      <creationDate>2017-06-21T14:36:34.091+0200</creationDate>
      <name>digitalCollection</name>
      <value>General</value>
    </property>
    <property id="118" container="1">
      <creationDate>2017-06-21T14:40:50.837+0200</creationDate>
      <name>Comment for depot status</name>
      <value />
    </property>
    <property id="117" container="1">
      <creationDate>2017-06-21T14:40:50.837+0200</creationDate>
      <name>Linking to external source</name>
      <value>http://goobi.intranda.com</value>
    </property>
    <property id="128" container="1">
      <creationDate>2017-06-21T14:44:25.089+0200</creationDate>
      <name>Opening angle</name>
      <value>180 degrees</value>
    </property>
    <property id="119" container="1">
      <creationDate>2017-06-21T14:40:50.837+0200</creationDate>
      <name>Placeholder card stored in depot</name>
      <value>true</value>
    </property>
    <property id="129" container="1">
      <creationDate>2017-06-21T14:44:25.089+0200</creationDate>
      <name>Reading direction is RTL</name>
      <value>false</value>
    </property>
  </properties>
  <templates />
  <workpiece />
  <tasks>
    <task id="61">
      <name>Data import</name>
      <priority>0</priority>
      <order>1</order>
      <status>3</status>
      <processingTime>2017-06-21T14:36:34.091+0200</processingTime>
      <processingStartTime>2017-06-21T14:36:13.019+0200</processingStartTime>
      <processingEndTime>2017-06-21T14:36:34.091+0200</processingEndTime>
      <user id="1" login="goobi">Administrator, Goobi</user>
      <editionType>4</editionType>
      <configuration useHomeDirectory="0" useMetsEditor="false" isAutomatic="false" readImages="false" writeImages="false" export="false" finalizeOnAccept="false" verifyOnFinalize="false" delayStep="false" updateMetadataIndex="false" generateDocket="false" batchStep="false" stepPlugin="" validationPlugin="" />
      <scriptStep scriptStep="false" />
      <httpStep httpStep="false" />
      <assignedUserGroups>
        <usergroup id="6" name="Project management" accessLevel="4">
          <role>Statistics_CurrentUsers</role>
          <role>Statistics_CurrentUsers_Details</role>
          <role>Statistics_General</role>
          <role>Statistics_Menu</role>
          <role>Statistics_Plugins</role>
          <role>Task_List</role>
          <role>Task_Menu</role>
          <role>Task_Mets_Files</role>
          <role>Task_Mets_Metadata</role>
          <role>Task_Mets_Pagination</role>
          <role>Task_Mets_Structure</role>
          <role>Workflow_General_Batches</role>
          <role>Workflow_General_Details</role>
          <role>Workflow_General_Details_Edit</role>
          <role>Workflow_General_Menu</role>
          <role>Workflow_General_Plugins</role>
          <role>Workflow_General_Search</role>
          <role>Workflow_ProcessTemplates</role>
          <role>Workflow_ProcessTemplates_Clone</role>
          <role>Workflow_ProcessTemplates_Create</role>
          <role>Workflow_ProcessTemplates_Import_Single</role>
          <role>Workflow_Processes</role>
          <role>Workflow_Processes_Allow_Download</role>
          <role>Workflow_Processes_Allow_Export</role>
          <role>Workflow_Processes_Allow_Linking</role>
          <role>Workflow_Processes_Show_Deactivated_Projects</role>
          <role>Workflow_Processes_Show_Finished</role>
        </usergroup>
      </assignedUserGroups>
    </task>
    <task id="62">
      <name>Get manuscript from book depot</name>
      <priority>0</priority>
      <order>2</order>
      <status>3</status>
      <processingTime>2017-06-21T14:40:59.451+0200</processingTime>
      <processingStartTime>2017-06-21T14:40:50.837+0200</processingStartTime>
      <processingEndTime>2017-06-21T14:40:59.451+0200</processingEndTime>
      <user id="6" login="testbookmanager">Fuller, Cedric</user>
      <editionType>1</editionType>
      <configuration useHomeDirectory="0" useMetsEditor="false" isAutomatic="false" readImages="false" writeImages="false" export="false" finalizeOnAccept="false" verifyOnFinalize="false" delayStep="false" updateMetadataIndex="false" generateDocket="false" batchStep="false" stepPlugin="" validationPlugin="" />
      <scriptStep scriptStep="false" />
      <httpStep httpStep="false" />
      <assignedUserGroups>
        <usergroup id="4" name="Book managing officers" accessLevel="4">
          <role>Statistics_CurrentUsers</role>
          <role>Task_List</role>
          <role>Task_Menu</role>
        </usergroup>
      </assignedUserGroups>
    </task>
    <task id="63">
      <name>Scanning</name>
      <priority>0</priority>
      <order>3</order>
      <status>3</status>
      <processingTime>2018-03-09T18:08:02.036+0100</processingTime>
      <processingStartTime>2017-06-21T14:44:25.106+0200</processingStartTime>
      <processingEndTime>2017-06-21T14:44:40.101+0200</processingEndTime>
      <user id="1" login="goobi">Administrator, Goobi</user>
      <editionType>3</editionType>
      <configuration useHomeDirectory="0" useMetsEditor="false" isAutomatic="false" readImages="false" writeImages="false" export="false" finalizeOnAccept="false" verifyOnFinalize="false" delayStep="false" updateMetadataIndex="false" generateDocket="false" batchStep="false" stepPlugin="intranda_step_fileUpload" validationPlugin="intranda_validation_filename" />
      <scriptStep scriptStep="false" />
      <httpStep httpStep="false" />
      <assignedUserGroups>
        <usergroup id="2" name="Scanning officers" accessLevel="4">
          <role>Statistics_CurrentUsers</role>
          <role>Task_List</role>
          <role>Task_Menu</role>
        </usergroup>
      </assignedUserGroups>
    </task>
    <task id="64">
      <name>Quality control</name>
      <priority>0</priority>
      <order>4</order>
      <status>3</status>
      <processingTime>2017-06-21T14:48:22.830+0200</processingTime>
      <processingStartTime>2017-06-21T14:48:16.487+0200</processingStartTime>
      <processingEndTime>2017-06-21T14:48:22.830+0200</processingEndTime>
      <user id="4" login="testqc">Readdie, Moss</user>
      <editionType>1</editionType>
      <configuration useHomeDirectory="0" useMetsEditor="false" isAutomatic="false" readImages="false" writeImages="false" export="false" finalizeOnAccept="false" verifyOnFinalize="false" delayStep="false" updateMetadataIndex="false" generateDocket="false" batchStep="false" stepPlugin="intranda_step_imageQA" validationPlugin="" />
      <scriptStep scriptStep="false" />
      <httpStep httpStep="false" />
      <assignedUserGroups>
        <usergroup id="3" name="Quality control officers" accessLevel="4">
          <role>Statistics_CurrentUsers</role>
          <role>Task_List</role>
          <role>Task_Menu</role>
        </usergroup>
      </assignedUserGroups>
    </task>
    <task id="65">
      <name>Image processing</name>
      <priority>0</priority>
      <order>5</order>
      <status>3</status>
      <processingTime>2017-06-21T14:48:23.411+0200</processingTime>
      <processingStartTime>2017-06-21T14:48:22.833+0200</processingStartTime>
      <processingEndTime>2017-06-21T14:48:23.411+0200</processingEndTime>
      <editionType>4</editionType>
      <configuration useHomeDirectory="0" useMetsEditor="false" isAutomatic="true" readImages="false" writeImages="false" export="false" finalizeOnAccept="false" verifyOnFinalize="false" delayStep="false" updateMetadataIndex="false" generateDocket="false" batchStep="false" stepPlugin="intranda_step_imageConverter" validationPlugin="" />
      <scriptStep scriptStep="false" />
      <httpStep httpStep="false" />
      <assignedUserGroups>
        <usergroup id="1" name="Administration" accessLevel="4">
          <role>Admin_Administrative_Tasks</role>
          <role>Admin_Dockets</role>
          <role>Admin_Ldap</role>
          <role>Admin_Menu</role>
          <role>Admin_Plugins</role>
          <role>Admin_Projects</role>
          <role>Admin_Rulesets</role>
          <role>Admin_Usergroups</role>
          <role>Admin_Users</role>
          <role>Admin_Users_Allow_Switch</role>
          <role>Plugin_Goobi_Exchange</role>
          <role>Statistics_CurrentUsers</role>
          <role>Statistics_CurrentUsers_Details</role>
          <role>Statistics_General</role>
          <role>Statistics_Menu</role>
          <role>Statistics_Plugins</role>
          <role>Task_List</role>
          <role>Task_Menu</role>
          <role>Task_Mets_Files</role>
          <role>Task_Mets_Metadata</role>
          <role>Task_Mets_Pagination</role>
          <role>Task_Mets_Structure</role>
          <role>Workflow_General_Batches</role>
          <role>Workflow_General_Details</role>
          <role>Workflow_General_Details_Edit</role>
          <role>Workflow_General_Menu</role>
          <role>Workflow_General_Plugins</role>
          <role>Workflow_General_Search</role>
          <role>Workflow_General_Show_All_Projects</role>
          <role>Workflow_ProcessTemplates</role>
          <role>Workflow_ProcessTemplates_Clone</role>
          <role>Workflow_ProcessTemplates_Create</role>
          <role>Workflow_ProcessTemplates_Import_Multi</role>
          <role>Workflow_ProcessTemplates_Import_Single</role>
          <role>Workflow_Processes</role>
          <role>Workflow_Processes_Allow_Download</role>
          <role>Workflow_Processes_Allow_Export</role>
          <role>Workflow_Processes_Allow_GoobiScript</role>
          <role>Workflow_Processes_Allow_Linking</role>
          <role>Workflow_Processes_Show_Deactivated_Projects</role>
          <role>Workflow_Processes_Show_Finished</role>
        </usergroup>
      </assignedUserGroups>
    </task>
    <task id="66">
      <name>Metadata enrichment</name>
      <priority>0</priority>
      <order>6</order>
      <status>2</status>
      <processingTime>2017-06-21T14:49:19.991+0200</processingTime>
      <processingStartTime>2017-06-21T14:49:19.991+0200</processingStartTime>
      <processingEndTime />
      <user id="5" login="testmetadata">Anson, Wilburn</user>
      <editionType>1</editionType>
      <configuration useHomeDirectory="0" useMetsEditor="true" isAutomatic="false" readImages="false" writeImages="false" export="false" finalizeOnAccept="false" verifyOnFinalize="true" delayStep="false" updateMetadataIndex="false" generateDocket="false" batchStep="false" stepPlugin="" validationPlugin="" />
      <scriptStep scriptStep="false" />
      <httpStep httpStep="false" />
      <assignedUserGroups>
        <usergroup id="5" name="Metadata officers" accessLevel="4">
          <role>Statistics_CurrentUsers</role>
          <role>Task_List</role>
          <role>Task_Menu</role>
          <role>Task_Mets_Metadata</role>
          <role>Task_Mets_Pagination</role>
          <role>Task_Mets_Structure</role>
        </usergroup>
      </assignedUserGroups>
    </task>
    <task id="67">
      <name>Export to viewer</name>
      <priority>0</priority>
      <order>7</order>
      <status>0</status>
      <processingTime>2017-06-21T14:36:13.019+0200</processingTime>
      <processingStartTime />
      <processingEndTime />
      <user id="1" login="goobi">Administrator, Goobi</user>
      <editionType>4</editionType>
      <configuration useHomeDirectory="0" useMetsEditor="false" isAutomatic="true" readImages="false" writeImages="false" export="true" finalizeOnAccept="false" verifyOnFinalize="false" delayStep="false" updateMetadataIndex="false" generateDocket="false" batchStep="false" stepPlugin="" validationPlugin="" />
      <scriptStep scriptStep="false" />
      <httpStep httpStep="false" />
      <assignedUserGroups>
        <usergroup id="1" name="Administration" accessLevel="4">
          <role>Admin_Administrative_Tasks</role>
          <role>Admin_Dockets</role>
          <role>Admin_Ldap</role>
          <role>Admin_Menu</role>
          <role>Admin_Plugins</role>
          <role>Admin_Projects</role>
          <role>Admin_Rulesets</role>
          <role>Admin_Usergroups</role>
          <role>Admin_Users</role>
          <role>Admin_Users_Allow_Switch</role>
          <role>Plugin_Goobi_Exchange</role>
          <role>Statistics_CurrentUsers</role>
          <role>Statistics_CurrentUsers_Details</role>
          <role>Statistics_General</role>
          <role>Statistics_Menu</role>
          <role>Statistics_Plugins</role>
          <role>Task_List</role>
          <role>Task_Menu</role>
          <role>Task_Mets_Files</role>
          <role>Task_Mets_Metadata</role>
          <role>Task_Mets_Pagination</role>
          <role>Task_Mets_Structure</role>
          <role>Workflow_General_Batches</role>
          <role>Workflow_General_Details</role>
          <role>Workflow_General_Details_Edit</role>
          <role>Workflow_General_Menu</role>
          <role>Workflow_General_Plugins</role>
          <role>Workflow_General_Search</role>
          <role>Workflow_General_Show_All_Projects</role>
          <role>Workflow_ProcessTemplates</role>
          <role>Workflow_ProcessTemplates_Clone</role>
          <role>Workflow_ProcessTemplates_Create</role>
          <role>Workflow_ProcessTemplates_Import_Multi</role>
          <role>Workflow_ProcessTemplates_Import_Single</role>
          <role>Workflow_Processes</role>
          <role>Workflow_Processes_Allow_Download</role>
          <role>Workflow_Processes_Allow_Export</role>
          <role>Workflow_Processes_Allow_GoobiScript</role>
          <role>Workflow_Processes_Allow_Linking</role>
          <role>Workflow_Processes_Show_Deactivated_Projects</role>
          <role>Workflow_Processes_Show_Finished</role>
        </usergroup>
      </assignedUserGroups>
    </task>
    <task id="68">
      <name>Bring manuscript back to book depot</name>
      <priority>0</priority>
      <order>8</order>
      <status>0</status>
      <processingTime>2017-06-21T14:36:13.019+0200</processingTime>
      <processingStartTime />
      <processingEndTime />
      <user id="1" login="goobi">Administrator, Goobi</user>
      <editionType>4</editionType>
      <configuration useHomeDirectory="0" useMetsEditor="false" isAutomatic="false" readImages="false" writeImages="false" export="false" finalizeOnAccept="false" verifyOnFinalize="false" delayStep="false" updateMetadataIndex="false" generateDocket="false" batchStep="false" stepPlugin="" validationPlugin="" />
      <scriptStep scriptStep="false" />
      <httpStep httpStep="false" />
      <assignedUserGroups>
        <usergroup id="4" name="Book managing officers" accessLevel="4">
          <role>Statistics_CurrentUsers</role>
          <role>Task_List</role>
          <role>Task_Menu</role>
        </usergroup>
      </assignedUserGroups>
    </task>
  </tasks>
</process>

The implementation of this part is the first step in exchanging data between two Goobi instances. With the help of this export file, the workflows can now be transferred to other systems.

https://github.com/intranda/goobi/commit/82ccf81197b0940c6a05b5de42a5422d131dbbc7 https://github.com/intranda/goobi/commit/0d49071fcdc3f7886d9a414adea60eac7a6fab66 https://github.com/intranda/goobi/commit/d46d92445797d7c6d3f853f41f70346b80e461a6 https://github.com/intranda/goobi/commit/a3da5bb9858822e16e9aba28733c7ee74dd39247 https://github.com/intranda/goobi/commit/55c13e61c8d2f9069eec0c0111e1683026fa7e70

Preparation for the Import of the extended process log for Goobi-to-Goobi Import

The work for exporting the workflow information into an XML file is only the first part for the future Goobi-to-Goobi data exchange. The second essential part for this is still in development. We are working intensively on a user interface that will allow you to import data from another Goobi into your own Goobi. The whole thing is implemented as an administration plug-in. However, the development has not yet been completed to the extent that an insight can already be given here. This will take place next month.

Bugfix for "broken pipe" log messages

In the past, the log messages from Goobi workflow often contained error messages with a reference to broken pipe. By changing the redirection after logging in, these error messages could now be reduced.

https://github.com/intranda/goobi/commit/755412a889fdd583ec7318272c2f2dcb593dcf9a

Core

Process properties

The process properties available in Goobi workflow can now also be displayed as a text area. They are configured using the familiar goobi_processProperties.xml. An example configuration is as follows:

    <property name="Beschreibung" container="0">
        <project>*</project>
        <showProcessCreation access="write" template="*" />
        <type>Textarea</type>
        <defaultvalue></defaultvalue>
    </property>

Snippets

• If the GoobiScript for deleting processes was called several times in succession using the same hit set, an error occurred which made it necessary to restart the application. The behaviour has been corrected.

• The default configuration of ActiveMQ has been adjusted so that the service only listens on localhost.

• An error that occurred when repeating certain variables in script steps has been fixed

Plugins

Import: Charité - Female doctors in the German Empire and persecuted doctors

A special import plugin was developed for the import of the two databases "Female Doctors in the German Empire and the Weimar Republic" and "Persecuted Doctors of the Berlin Municipal Health Service (1933-1945)" and the conversion into METS/MODS files.

Step: Metadata per image

A very powerful step plugin was finalised this month. This allows a wide range of metadata and links to be recorded for each image in a process.

Step: Duplicate tasks

The plugin makes it possible to read a task property and duplicate an existing workflow step multiple times for each line in the property. It also duplicates the properties split into individual properties per line.

Step: Extract PDF files

The already known plugin for the fully automatic preparation of PDF files has been supplemented by free file naming and updated documentation.

Code analysis

The following screenshot shows the SonarCloud analysis of the current release. More information is available directly on the project page.

Version number

The current version number of Goobi workflow with this release is: 23.11.1. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.11.1</version>
</dependency>

Core

Usability

We are always looking for small areas in the user interface in order to specifically improve usability. In August, it was the area of actions within an assumed task, where we tried to create clarity simply by adapting the terminology used.

Here are two screenshots of the area before the update:

There was confusion mainly because the first tab was called "Execute task" and the visualisation suggested a button. However, nothing happened when you clicked on it. The second confusion arose in the area of errors and correction messages, where a "correction requested" and an "error solution reported" were displayed. The expectation here was rather that you would "report an error" and "send a correction". The user interface was made clearer simply by adapting the corresponding translations. Here is the result:

GoobiScript

The GoobiScript openNextStep has been added to set the first locked task of a process from the hit set to open if no other task is open. If it is an automatic task, it is also started automatically.

In addition, the addStep GoobiScript now has the option of using the number: end setting to insert the task at the end of the process, even without knowing the exact number. This is determined automatically during the setting.

REST

A new endpoint has been added to the REST API to start open automatic tasks.

In addition, the endpoint for creating new processes has been extended so that metadata and properties can optionally be added at the same time.

Plugins

Export: VLM

In addition to a password, the VLM export now also supports the option of using the SFTP functionality with an SSKH key.

Export: Heris

A special plugin for the export of five selected images and metadata as a JSON file was developed for the Federal Monuments Office in Austria.

Import: Nuremberg House Books

A special import plugin was developed for the import of the Nuremberg house books and the conversion of an SQL data structure into METS/MODS files.

Step: Image selection

A step plugin for selecting and defining a sequence of images and saving the result for further processing as a process property was finalised.

Code analysis

The following screenshot shows the SonarCloud analysis of the current release. More information is available directly on the project page.

Version number

The current version number of Goobi workflow with this release is: 23.08. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.08</version>
</dependency>

Coming soon

• Documentation of step details

• Personal API Token

Core

Error and correction messages

Error and correction messages can be sent in an accepted task. New is that now an optional dropdown menu is available where a list of classic error types can be configured.

The configuration is done for both error and correction messages in the central goobi_config.properties file. Here is an example configuration:

# Text templates for error reporting and problem solutions
task.error.Missing\ pages=The following pages are missing: {}
task.error.Blurred\ images=The images {} are unsharp. Please create these again.
task.solution.Problem\ solved=The problem was solved. {}
task.solution.The\ original\ print\ is\ blurred=The original pages are printed blurry. It is not possible to create sharper images. {}

Mail sending

Mail sending from Goobi workflow has been unified and different plugins that can also send emails can now access the same configuration. This eliminates redundancies in the code, minimizes maintenance and makes configuration easier.

Code quality

For some time, we have been working in Goobi workflow with Sonarcloud for static analysis of code quality. In July last year, we started with a good 4,600 issues noted. With this release we have reached about 1,600 issues. In the last month alone, we were able to process around 1,000 issues. In addition, around 75 unit tests were transferred to the core.

Performance

When creating and deleting processes with a lot of steps (let's say more than 50) there were performance problems. These problems could be solved by changing SQL calls and Java code.

Plugins

Import: Excel import

The Excel import plugin now allows the specification of a replacewith attribute for the generation of task titles - analog to the configuration in goobi_projects.xml - in order to replace spaces with underscores, for example.

Step: PDF generation

The PDF generation step plugin now allows specifying and using the PDF generation variants recently introduced in ContentServer. This allows, for example, valid PDF-A files to be generated as a step within a production template and stored in a specified location.

Code analysis

The following screenshot shows the SonarCloud analysis of the current release. More information is available directly on the project page.

A few words about the failing analysis:

In SonarCloud, quality requirements, called quality gates, can be defined. The quality gates are applied once to the entire source code and once to the source code that has been modified in the last 30 days.

The now failing quality gate in Goobi workflow refers to the adapted source code in the last 30 days. There, a test coverage of at least 25% is required and the duplications should be below 10%. Since we have touched a lot of source code with this release, we do not meet these requirements - based on the source code of the last 30 days.

However, the quality gate for the entire source code is met.

Version number

The current version number of Goobi workflow with this release is: 23.05.1. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.05.1</version>
</dependency>

September 2023

September

Core

Usability

The Goobi workflow creation screen has always prevented a process with the same name from being created again. You would then receive a short error message saying that the process already exists. This has been enhanced so that you can now also view the details of the existing process.

GoobiScript

The GoobiScript calls metadataAdd and metadataReplace can now also create or manipulate authority data values. This makes it possible, for example, to subsequently adjust the GND ID for persons in bulk.

At the same time, the GoobiScript metadataReplaceAdvanced was introduced. This call also allows you to work with regular expressions.

REST

September also brought changes to the REST interface, two new endpoints were added. The first allows a search and returns a list of process IDs:

The second allows you to complete tasks with a defined title:

Plugins

Import: Excel to EAD

An import plugin has been developed to create an EAD archive structure with metadata from an Excel file.

Coming soon

• GoobiScript templates

• Documentation of step details

Core

API Token

In the settings of one's own user account, there is a new tab in which an API token can be generated and equipped with the corresponding rights.

Only super administrators or user accounts that belong to a group to which the corresponding right has been assigned have the right to create a token.

This token can then be used with a basic auth to query the previously enabled commands.

Create process

When creating a task, the mask can be configured via the file goobi_projects.xml. Multiselect fields could also be defined with the attribute multiselect="true". If several values were then selected, they ended up one after the other in the configured metadatum or the defined property.

As of this release, a separate metadatum or property is written for each value.

Plugins

Step: OCR to metadata

A new step plugin has been developed that allows to convert the OCR text of all images of a process into a metadatum.

Snippets

• ALTO editing in the metadata editor can now also save files in S3.

• The plugin for automatic PDF extraction has received minor improvements, for example in the case of missing spaces in the full text.

Coming soon

• Documentation of step details

• Improved PDF-A generation

Core

User interface

We have worked several hundred hours in the background on it and in the result only little changes at the surface first of all... In the substructure, however, a lot: Where source code was copied and modified over time, there is now a unified system again. New structures have been built, redundancies removed, code and markup for the same things unified and thus the whole structure simplified.

Probably the first thing you'll notice is that everything is a bit more airy and here and there some things might even look a bit strange now. Unusual in any case. But the basic structure and also the design remains the same.

Based on this work, we can now start to work on the usability: How do we want to arrange and name buttons and where? Where do we speak of the terminology in the plural and where singular? Which boxes are displayed where, what is the order of information, and how do we highlight relevant things, hide irrelevant or incidental ones, or reduce the focus? What is a consistent and reliable path to a goal and where do we want to deliberately create shortcuts? With this basis, we can change many things in one central place and it automatically affects all areas. Before we would have had to look at very very many places individually.

Please report things that now look funny or are incomprehensible.

REST API

In April, as part of a project, we began to revisit the REST API and develop what we believe to be essential functionality, refactoring and consolidating existing code as well. Specifically, we considered tasks, steps, properties and the journal and implemented the following functionalities:

• Create tasks, read information, update or delete tasks.

• Read out, update, add, delete or execute steps for an operation

• Read journal and add, update or delete records

• Read out, add, edit and delete properties

The logic uses - where possible - existing code from GoobiScript. The terminology is also chosen to be identical, so there should be recognition effects when working here. The new code has been given meaningful test coverage and the REST commands are also all listed and described in the Swagger documentation.

The new REST endpoints allow integration of functionality with external systems via a standardized interface in the Goobi workflow core.

Here are four example calls for the possibilities available for the journal:

Plugins

For some time now, the plugins installed in an instance can be viewed via the interface. On the page you can also see if the plugin has been tested against the installed Goobi workflow version number.

With the April release, we started to not only test the plugins for compatibility, but also tag them with an identical version number. This way it is now also possible to immediately identify in the plugin repository the commits that ensure compatibility to a specific core version.

We still need to observe the solution we have found now and possibly also readjust it, but we have already come a big step closer to our goal: "To have plugin versions analogous to the Goobi workflow Core".

Code analysis

The following screenshot shows the SonarCloud analysis of the current release. More information is available directly on the project page.

Version number

The current version number of Goobi workflow with this release is: 23.04.3. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.04.3</version>
</dependency>

Coming soon

• Porting of user interface to new framework

• Documentation of step details

• Improved PDF-A generation

• REST API

Core

Periodic tasks

For some time now, Goobi workflow has been able to trigger certain tasks periodically. There are internal tasks that are executed once a day, but also plugins like DataPoller or delay plugins. This functionality works stably and reliably, but had two limitations that we addressed with this release.

First, everything was done in secret. There was no overview of which regular tasks existed, when they ran and if there was an error or not. For this overview, there is now a new page in the "Administration" section that lists the information mentioned. Also an individual execution or pausing of tasks is possible there.

The second part was the configuration options. Either things were not configurable at all or they were given in seconds since midnight. To start something at 15:30 then the value 55800 had to be specified. Not exactly intuitive. Now the cron syntax is used, which is much more intuitive to read and also more powerful in configuration. The syntax for 15:30 every day is now 0 30 15 * * ?

Sorting for additional columns

For some time now, it has been possible to configure additional columns for the table on the "Tasks" and "My tasks" page in the settings of a user account. From now on these are also sortable if they are in the database. This is for example the case for variables like {db_meta.XYZ}, {process.XYZ}, {product.XYZ} or {template.XYZ}. Other variables like {origpath} or even {meta.XYZ} are not sortable.

Usability

The usability issue is progressing in small steps. This month we have taken a look at the project configuration and there especially at the tab "METS Parameters" which has been renamed to "Content settings". The names of the lines as well as the inline help have been massively revised and provided with useful examples.

Snippets

• In the queues, the error table is now automatically and regularly emptied.

• In the project settings, a custom URL for the DFG Viewer METS file can be stored in the "Content settings" area.

Coming soon

• Porting of user interface to new framework

• Periodic tasks

• Documentation of step details

Core

ALTO Editor

The ALTO Editor for OCR correction, which can be opened in the Metadata Editor if required, now also supports right-to-left languages. Switching is done automatically based on the selected setting in the Metadata Editor itself.

Usability

Minor usability changes are gradually being incorporated into the interface. In this digest we would like to go into a tiny sub-area and show what this means in practice. The starting point in this case is the metadata editor and the possibility to add another metadata. So far this looked like this:

We have looked at this small sub-area and noted the following points:

• The heading "Create new metadata" suggests by the plural that one can "create" here several at the same time. But this is not possible because in the mask only one can be "created" at a time.

• The word "create" is linked by the terminology rather with "create process". The word "add" is more appropriate here, since a new metadatum is to be added to an existing process.

• Also, the word "new" in the context is actually too much. "Add metadata" is also clear and unambiguous. At this point, the ruleset is not edited.

• "Type" is more likely to be associated with something like "Text, MultiSelect, DropDown, Textarea", but not "Additional title". A metadata of type "Additional title" is weird. Better is simply "Name".

• "Save the changes" says that you have changed something. But something is "created" or isn't "Add" the correct terminology here?

The result of the considerations looks as follows:

Of course, there is much more you can ask yourself at this point:

• Why is the box collapsible and expandable?

• From a UX perspective, is it right to have the cancel and add buttons equally weighted visually?

• Why isn't this simply opened in a modal as an overlay?

• ...

But these questions are more complex to answer and we are taking small steps. This example is just to show how we make things better in various places and little by little.

Plugins

Step: Metadata editing

The step plugin to edit metadata directly from the accepted task can now hide empty fields. This was not possible before and led to very long lists in use cases where the plugin was used to quickly check metadata.

Version number

The current version number of Goobi workflow with this release is: 23.01. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.01</version>
</dependency>

General

Core

Config Editor

The former plugin for the Goobi workflow configuration editor has been integrated into the core and is available from this release without additional installation.

Frontend build system

The interface of Goobi workflow is significantly styled using the classic web technologies CSS and Javascript. However, development takes place in SASS or LESS files, for example, which are then compiled into CSS. There are also various individual files in JavaScript, which are then combined into one file for the application. In the past, compiling and summarising took place on the developer's PC, who then checked the finished files into the repository. From the February release onwards, these steps will be carried out automatically in the CI environment. This simplifies the development process and reduces the potential for merge conflicts.

Snippets

• The comparator for file names has been fundamentally revised so that ascending and descending sorting of files in the core and in plugins is easier and more error-free.

Plugins

Export: VLM

The VLM export plugin can now use different configuration blocks based on metadata and conditions.

Step: Rename files

The step plugin for renaming files has been massively extended and can now contain several <condition /> and <replace /> arguments in the new configuration section <namepart />.

Step: Metadata edition

There was a bugfix in the step plugin for editing metadata to save information about the representative correctly.

Code analysis

The following screenshot shows the SonarCloud analysis of the current release. More information is available directly on the project page.

Version number

The current version number of Goobi workflow with this release is: 24.02.2. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>24.02.2</version>
</dependency>

Coming soon

• Porting of user interface to new framework

• Periodic tasks

• Documentation of step details

Core

Performance

A bottleneck in the performance of SQL queries with "Like" queries could be identified and fixed if. This was the case, for example, when queries for step titles were executed. Loading times could be reduced from 155 seconds to 3.2 seconds in some cases.

A significant speed increase was also achieved in the area of automatic tasks by changing the SQL queries.

The result is a faster responding user interface in certain situations.

Search

Users can now configure individual search entries. For this purpose, a new input area for individual search settings is available in the user settings in the "General" area.

Each line is structured according to the name=search filter principle where the placeholder {} stands for the search term. An example configuration could be as follows:

Default={}
Eigenschaft={} |processproperty:Template:{}
Projekt={} |project:{}
mit Leerzeichen="{}" "|processproperty:Material type:{}" 
Metadaten={} |meta:TitleDocMain:{}

If configured, a drop-down menu with the configured values appears in front of the search slot above the table on the "Operations" page.

Plugins

Step: Rename files

The file renaming plugin used to rename automatically in all task folders. Now it can also work only in certain folders. The use case is that one has different upload steps, the files are stored in each case in a temporary folder, then the plugin names the files individually and these are then moved with a script into the _master folder. This way you can ensure an individual and yet uniform naming of the files.

Step: Change workflow

The plugin for changing the workflow based on task properties can now also influence the priority for individual or all steps of a task. The background to this enhancement is that within a workflow the decision is made whether the task has to be put on the fast lane or not. If overtaking lane, then all steps in the task get a higher priority and appear accordingly higher up in the tasks.

Version number

The current version number of Goobi workflow with this release is: 23.02. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>23.02</version>
</dependency>

Core

Saving in the metadata editor

Within the metadata editor, we noticed that different browsers reacted differently when users clicked on the save button. Specifically, the Chrome and Safari web browsers did not perform any function when the Save button was clicked, while Firefox, for example, worked correctly.

Incidentally, the button with the title 'Save and return' was not affected by this problem..

After some back and forth and a lot of research into the causes, we were finally able to solve the problem and provide a fix with the release.

New journal functionality

For 8 years now, Goobi workflow has offered the possibility of tracking various information and communications via the process log. It has also been possible for some time to upload files (e.g. for restoration documents). This was expanded about 1.5 years ago to include the option of saving damage documents when creating processes.

In other words: the process log is simply practical. So practical that we wanted to have it in other places as well. To do this, we had to foolishly rename it so that it would no longer be used only for processes.

The result is the journal. Functionally, it has remained the same as for the processes. Now, however, it is also available for the institutions and, at least in one special case (delivery of deposit copies), also for users.

We can think of other places in Goobi workflow where the journal could be helpful (e.g. for projects). Perhaps there will be an opportunity to do this soon, because someone else would find it just as practical. In that case, we will be able to expand the journal and report back here.

Accelerated loading behaviour for extensive works

Especially in the case of very extensive works, there were always enormous delays in the loading speed when using S3 as a storage system. The reason for this is that an S3 system has no folder hierarchy and all files instead have enormously long file names. This looks like this, for example:

s3://bucket/123/images/bla_media/00000001.tif
s3://bucket/123/images/bla_media/00000002.tif
s3://bucket/123/images/master_bla_media/00000001.tif
s3://bucket/123/images/master_bla_media/00000002.tif
s3://bucket/123/images/bla_jpg/00000001.tif
s3://bucket/123/images/bla_jpg/00000002.tif

This deviating structure meant that when determining the master directory, for example, all files always had to be run through in order to check whether the directory name being searched for was represented. Especially with very extensive works, performance suffered greatly.

We have now changed this so that when searching for folders in the desired hierarchy level, everything after the next slash ( /) is no longer taken into account and so when searching within the images folder, significantly fewer file names to be checked are listed.

In addition, we have now always limited the file search to the actually relevant folder, so that even when searching for the master images, the data of all folders of a process are no longer queried.

Another big improvement in performance resulted from the fact that in the IIIF API we now no longer check per image whether a directory with the name 'thumbs' exists, but cache the result for a while. This has also allowed us to significantly reduce the number of requests to the system once again.

Blemish with empty table columns removed

During the detailed display of workflow information of individual work steps, it occasionally happened that empty table columns were displayed.

We have checked this once and revised it. This means that this small display error no longer occurs.

Better check of the available storage space

We noticed that the metadata editor did not display a user-friendly message if there was not enough storage available on the Goobi server to save the metadata while editing the metadata. Ideally, of course, this should not happen at all in everyday life. However, if it does happen, Goobi now notices the lack of storage space better and displays an easily understandable message for the user.

Interaction with a proxy improved

If it is necessary for Goobi itself to communicate with other services via a proxy, this was already possible quite easily in the past. However, a new requirement has arisen due to the increasing use of archive management. This requires communication with another local database, which in turn does not require communication via a proxy.

With this version, therefore, a new configuration option has been added, which from now on allows Goobi not to use the proxy for communication with its own server (localhost). This is configured with the new switch as follows:

http_proxyIgnoreHost=localhost

Extended LIDO support

If Goobi is used to use the LIDO format instead of METS, especially in the case of museum objects, from this version onwards it is possible to write empty elements within the generated XML files. For this purpose, a new configuration option has been created for the rule sets, internally specifying the following there:

<exportEmptyNode>true</exportEmptyNode> 

Illustrated by a concrete example, this looks like this:

<Metadata>
    <InternalName>OtherTitle</InternalName>
    <exportEmptyNode>true</exportEmptyNode>
    <XPath>./lido:objectIdentificationWrap/lido:titleWrap/lido:titleSet/lido:appellationValue[@lido:pref='alternative']</XPath>
    <WriteXPath>./lido:objectIdentificationWrap/lido:titleWrap/lido:titleSet/#lido:appellationValue[@lido:pref='alternative']</WriteXPath>
</Metadata>

The result within the generated LIDO would then look like this:

<lido:objectIdentificationWrap>
    <lido:titleWrap>
        <lido:titleSet>
            <lido:appellationValue lido:pref="preferred">Testdatensatz_LIDO, Zusatztitel</lido:appellationValue>
            <lido:appellationValue lido:pref="alternative" />
        </lido:titleSet>
    </lido:titleWrap>
</lido:objectIdentificationWrap>

The configuration made here has the following influence:

If the metadatum in question exists, it is exported normally. If, on the other hand, it does not exist, it is checked whether it is permitted in the current structure element. If this is the case, 'WriteXPath' is also executed. Such a configuration can also be specified for a group or for a metadatum in a group.

If such a configuration is not made, this line is omitted in the generated LIDO:

<lido:appellationValue lido:pref="alternative" />

More on the configuration options for the metadata exports can be found in the following documentation:

Plugins

Plugin for editing personal data nears completion

Again this month, a lot of energy has gone into the completion of a new plugin for the granular collection of personal data. In addition to various minor and major bug fixes, the user interface, among other things, was once again thoroughly checked and adapted.

We expect to complete the plugin in October and to be able to document it thoroughly then. We will keep you up to date.

Extension of the Catalogue Poller

The Catalogue Poller is used to automatically update the metadata within the METS files periodically for numerous processes. In this way, catalogue changes can be automatically adopted and are subsequently also taken into account for the presentation of the digital copies. This plugin has now been fundamentally revised and equipped with further functions. In addition to some changes to the user interface, the main addition here was the new possibility that the Catalogue Poller can now also carry out test runs, which make it possible to see what changes would be made to the data without actually changing it. The result of such a test run can be downloaded as an Excel file.

Also new is the possibility that within the configuration of the metadata to be taken into account, it is no longer possible to name only those fields that are not to be changed (blacklist). Instead, it is now also possible to name only those fields that are to be changed (whitelist). Here is an excerpt from the configuration:

      <!-- fieldList: Must have a mode attribute which can contain either blacklist or whitelist as a value.
        blacklist: All fields are updated except the defined ones. This is a potential dangerous setting!
        whitelist: Only the definied fields are updated. All others are skipped. 
        field: Use the internal metadata names from the ruleset as field definition
      -->
      <fieldList mode="blacklist">
         <field>viewerinstance</field>
         <field>singleDigCollection</field>
         <field>pathimagefiles</field>
         <field>_urn</field>
         <field>_representative</field>
      </fieldList>

The documentation of the plugin can be found in an updated form at the following address:

The plugin can be downloaded as usual from GitHub here:

Plugin for the generation of PDFs extended

For some time now, there has been a plugin that can automatically generate PDFs during workflow execution. Until now, however, its configurability was limited. This has improved with the new version, so that an individual configuration can be defined for each project and title of a work step. This can look like this, for example:

<?xml version="1.0" encoding="UTF-8"?>
<config_plugin>
    <!-- order of configuration is: 
         1.) project name and step name matches 
         2.) step name matches and project is * 
         3.) project name matches and step name is * 
         4.) project name and step name are * 
    -->
    <config>
        <!-- which projects to use for (can be more then one, otherwise use *) -->
        <project>*</project>
        <!-- which stepss to use for (can be more then one, otherwise use *) -->
        <step>*</step>
         <!-- Choose the source images folder that shall be used for PDF generation. Possible values are 'media' and 'master' -->
        <imagesFolder>media</imagesFolder>
        <!-- If enabled, PDF files for every single image are generated -->
        <pagePdf enabled="true" /> 
        <!-- If enabled, a PDF file for the entire record is generated. This file includes the table of contents from the METS file if exists. -->
        <fullPdf enabled="true" />
    </config>
    
    <config>
        <project>testocr</project>
        <step>testpdf</step>
        <imagesFolder>master</imagesFolder>
        <pagePdf enabled="true" />
        <fullPdf enabled="false" />
    </config>
</config_plugin>

In this way, the plugin can also be operated several times in a Goobi in different projects and workflows and with different settings in each case.

New REST plugin for MARC data reference

In the context of our developments for electronic deposit copies, there was a requirement that it should be possible to extract data from selected processes in Goobi workflow from external systems. The following functionality is available for this purpose:

Within the workflow, there is first a step for MARC export, which exports the metadata as marcxml. The following plugin is used for this:

The exported data then ends up in a directory that is monitored by the new REST plugin. This is the following plugin:

Within this REST plugin, various commands are available to enable communication between Goobi workflow and a library catalogue.

The following commands are available:

List all importable data

This command lists all records that are currently available.

Calling the REST API:

curl -s -H "Content-Type: application/xml" -H "token:SECRET" http://127.0.0.1:8080/goobi/api/delivery/listfiles

Response of the REST API:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<files>
    <file>a643b394-d56c-415c-9da9-b07008187445.xml</file>
</files>

Query a specific data set

This command can be used to query a specific data set.

Calling the REST API:

curl -s -H "Content-Type: application/xml" -H "token:SECRET" http://127.0.0.1:8080/goobi/api/delivery/get/a643b394-d56c-415c-9da9-b07008187445.xml

The REST API then responds with the queried MARC data set.

Successful data transfer

This call confirms the successful transfer of the data set and reports the catalogue ID (aDIS ID). The step is then closed.

Calling the REST API:

curl -s -X POST -H "Content-Type: application/xml" -H "token:SECRET" http://127.0.0.1:8080/goobi/api/delivery/finish/a643b394-d56c-415c-9da9-b07008187445.xml/aDIS-ID12345

Reporting a Data Transfer Error

This command is used to report a data set to Goobi whose data transfer was not successful. The error is reported and the work step is set to error status.

Calling the REST API:

curl -s -X POST -H "Content-Type: application/xml" -H "token:SECRET" -d'<msg><type>error</type><message>Corrupt filename</message></msg>' http://127.0.0.1:8080/goobi/api/delivery/error/a643b394-d56c-415c-9da9-b07008187445.xml

With the last two calls, a workflow step is closed or set to error status. The MARC data set is then removed from the export directory so that it is not listed again in the next search.

Documentation

Update instructions for the XMP plugin added

We noticed that the update instructions in July 2020 were still missing a note for the necessary adjustments to the configuration of the XMP header plugin. We have now added these so that everything should be up to date.

As before, the update instructions can be found at the following address:

Version number

The current version number of Goobi workflow with this release is: 22.09. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>22.09</version>
</dependency>

Core

Bessere Anzeige von Validierungsergebnissen im Metadateneditor

In Goobi workflow, the rule set can be used to control which metadata may occur, for example, how often, where they are permitted and what content is possible in them. In this way, a quality requirement for the metadata to be captured can be well controlled and adhered to. In order for the user to comply well with these requirements, the metadata editor displays validation messages that indicate problems. In the past, however, these messages were only ever displayed globally, not directly on the metadata in question itself and also not on the associated structural element. In other words: As a user, you were a bit perplexed as to which field was not valid and where exactly. You had to do some searching first.

From now on, there are some major changes at this point. Validation messages appear directly at the field in red colour and point out the problems directly with a helpful message.

And also in the tree display of the structural elements in the left-hand pane, the validation problems are now highlighted and now allow the user to find very quickly the metadatum in question for which there is a message.

If you hold the mouse pointer over a structural element with errors, you will immediately see the corresponding message in a pop-up.

By the way, we also took the opportunity to introduce a new key combination for performing the validation. It is: CTRL + SHIFT + V.

Search slot in the task list finally conveniently usable again

After minor optical adjustments the search slot within the task list of Goobi workflow had become too small, it was a bit too cumbersome to use lately. It was simply no longer possible to keep track of which search terms had already been entered when the search query became a little longer.

As of this month, this circumstance is a thing of the past. The search slot is now basically much longer on regular displays.

And even on mobile devices, the usability is such that you can work reasonably with the search:

Correction of URLs for vocabulary records

Goobi workflow has had a vocabulary management system for some time, which can be used, among other things, to make the metadata editor as convenient as possible by means of selection fields with controlled lists. It is also possible that the values are not only available in a selection field but can also be selected from a hit list after using a search mask.

In such a case, not only the selected value is taken over for the subsequent display of the digitised material in the Goobi viewer. Instead, a URL for linking to the corresponding vocabulary record is also entered.

Unfortunately, a small error had crept in here, so that these URLs may not have been correct. We have now corrected this behaviour so that the URLs are now correct. However, please check whether you have already used this functionality. In this case, the URLs you have already entered should be corrected.

First layout changes go live

We had already announced that we would be making some major changes in the background with regard to accessibility and usability. In the case of the administration screen for users, we are now slowly putting these changes into operation. As announced, this will be accompanied by minor visual changes. This should soon have an impact on other areas as well.

This is how it already looks:

GoobiScript now also for changing metadata within metadata groups

For some time there have been various GoobiScripts for changing data within metadata. Unfortunately, until now this was not easily possible for metadata groups. There have been some changes this month so that the existing GoobiScripts can also be used for metadata groups if the parameters are set correctly.

Since the changes are quite extensive, we have also adapted and extended the documentation. As before, it can be found at the following address:

Calls of scripts in the journal better traceable

Within the journal (formerly process log), the individual activities of a workflow can usually be easily traced. Among other things, the calls of scripts were already logged there in the past in order to be able to trace them better in the event of problems. What was missing, however, was that the variables used were also displayed in their replaced form within the script calls. This has now been adapted so that after each call of a script within the journal, both the script call in its original form and in its resolved form are listed.

In the case of a simple script like this one:

... this is what the call within the journal now looks like:

By the way, we also took the opportunity to adjust the height of the box so that it makes better use of the available space and you can see more information at once without having to scroll.

Super admins now always have all rights

Goobi workflow has a fairly granular rights system. It is so flexible that even plugins can define their own rights that a user must have in order to work with the plugin.

To make life a little easier for the administrators (including ourselves), we have now introduced that all users who have the status of 'super administrator' automatically have all rights in Goobi, including the individual rights that plug-ins bring with them.

Plugins

New plugin for flexible file validation

As part of our developments in the delivery of electronic deposit copies, we needed a flexible way for files uploaded by users to be validated and checked for compliance with requirements. Among other things, it should be possible to use any external tools for validation and to ensure compliance with defined levels.

The result of the development is an extremely flexible validation plugin that can be configured very granularly. We have published extensive documentation for the plugin here, including several useful configuration examples:

And as usual, the source code of the plugin and a compiled version can be found on GitHub:

Documentation

URN plugin now allows use behind a proxy

This should only affect a few users, but for those it is important:

The still quite new plugin for URN registration could not be used until now if the Goobi server is only allowed to access the internet via a proxy. Here we have now made an adjustment that from now on also allows operation behind a proxy.

We have updated the documentation for this change:

The plugin can be found as usual here on GitHub:

Documentation for the Artist Dictionary Plugin is online

After a long development on the plugin for capturing linked entities, we have now also completed the related plugins and published them on GitHub. On the one hand, this is a special dashboard plugin:

On the other hand, this workflow plugin is also required:

The installation and configuration of these plugins are now also fully and very granularly documented. They are available as usual on our documentation platform:

Version number

The current version number of Goobi workflow with this release is: 22.10. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>22.10</version>
</dependency>

Projects

easydb

Together with a partner we developed a process chain that allows to import records exported from an easydb fully automatically into Goobi workflow, to process them and to export them into a Goobi viewer. For this process chain three plugins were used and extended to meet the requirements of the project.

The generic XML OPAC plugin can now not only query a remote API, but also read XML files from the file system.

The DataPoller (formerly CataloguePoller) has been extended as part of the project to not only work over a filter set of tasks, but also iterate over the files in a folder. Also new is the functionality for individual work orders to be processed through the internal ticket queue.

The Fetch-Images-From-Metadata step plugin has undergone minor adjustments to, for example, find files without file extensions, deal with duplicate file names, or skip error messages for missing images.

The process chain also works periodically, creating new operations or updating existing ones. Once imported, records proceed through the workflow steps as usual.

Core

Plugins in the metadata editor

The core of Goobi workflow has been extended to offer plugins within the metadata editor. These plugins are integrated in the blue header at the top and provide their functionality.

The decisive factor for the development was the requirement to be able to change the publication type within the metadata editor. For this, template processes should be created and certain, marked metadata should be taken over during the change.

Metadata generation

It happens again and again that certain metadata are to be formed automatically from other values. An example could be the main title of a letter, which should always be generated as follows: "Letter from [sender] to [recipient] at [place], [date]".

To provide this functionality in the metadata editor the configuration file goobi_metadataDisplayRules.xml was extended by the possibility to generate values from other metadata values for certain metadata under configurable conditions.

A configuration for the above example could look like this:

<generate ref="TitleDocMain">
    <condition>goobi:metadata[@name='genre'][text()='Letter']</condition>
    <value>Letter from [ACTOR_FROM] to [ACTOR_TO] at [PLACE], [DATE]</value>

    <item>
        <label>ACTOR_FROM</label>
        <type>xpath</type>
        <field>goobi:metadata[@type='group'][@name='name'][goobi:metadata[@name='name_role_roleTerm'][text()='is created by (intellectual)']]/goobi:metadata[@name='name_displayForm']</field>
        <regularExpression>(.+), .*</regularExpression>
        <replacement>$1</replacement>
    </item>
    <item>
        <label>ACTOR_TO</label>
        <type>xpath</type>
        <field>goobi:metadata[@type='group'][@name='name'][goobi:metadata[@name='name_role_roleTerm'][text()='is intended for']]/goobi:metadata[@name='name_displayForm']</field>
        <regularExpression>(.+), .*</regularExpression>
        <replacement>$1</replacement>
    </item>
    <item>
        <label>PLACE</label>
        <type>xpath</type>
        <field>goobi:metadata[@type='group'][@name='location_related_place'][goobi:metadata[@name='location_holdingExternal_relation_type'][text()='is bound from']]/goobi:metadata[@name='location_physicalLocation_related_place']</field>
    </item>
    <item>
        <label>DATE</label>
        <type>variable</type>
        <field>{meta.originInfo_dateCreated_start}</field>
        <regularExpression>([0-9]{4})-[0-9]{2}-[(0-9]{2}</regularExpression>
        <replacement>$1</replacement>
    </item>
</generate>

PDF download in the metadata editor

Within the metadata editor there is now the possibility to generate a PDF of all images at any time. For this purpose there is a new PDF button in the left column with the structure tree

Plugins

Step: Rename process

There is a new step plugin to rename a task within a workflow. For this purpose, the variables available in Goobi workflow can be used. Additionally underscores and minus are allowed as characters. Empty variables are skipped and spaces are removed.

Export: Newspaper export

The plugin for exporting to the DDB newspaper portal has been adapted for better display of data in the DFG Viewer.

Documentation

The documentation of goobi_config.properties has been extensively revised, documented and is now available in chapter 7.2 of the documentation:

Version number

The current version number of Goobi workflow with this release is: 22.11. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>22.11</version>
</dependency>

Core

General layout changeover to Bootstrap 5

We already announced it. Some users may have already noticed that the layout will change minimally (at first): We are gradually changing the layout of Goobi workflow to Bootstrap 5. The first pages have already been moved.

But who would have thought it: it is more elaborate than expected. So it will continue to accompany us as a theme for a bit.

Cleaning up the display of active sessions

We have already made some changes to the login and logout area in recent months, including the display of active users and logging. However, we noticed that after logging out, the user's session was not completely cleaned up immediately. We have adjusted a few things here and then put them through their paces. We are now satisfied with this, the storage space for the sessions is now somewhat smaller and, from our point of view, it can stay that way.

Database connection for production templates optimised

On some Goobi systems, especially those with a large database, it was noticeable that the listing of production templates was significantly slower than the listing of tasks. We took a closer look at this this month and were able to determine that the more extensive rights check of the project memberships had a significant influence here. Fortunately, this could be remedied quite well, so that now no performance losses should be noticeable in the production templates.

Password length check corrected

Unfortunately, we noticed a small bug concerning the checking of the password length of users. The length of the password was incorrectly checked - and noted as too short - even if data of an existing user was to be changed without a field for password entry being displayed in the user interface at all. We have corrected the error in this release.

Automatic unit tests

To ensure the quality of software, software developers can implement automatic tests. Such tests can then automatically check individual functionalities of the overall software and are run at least before a release is published.

Goobi workflow has had several hundred such tests for many years. Nevertheless, it is our goal to cover as much of the source code as possible with tests in order to ensure code quality on the one hand, and on the other hand, to always be sure that the existing functionality is still guaranteed in the case of new developments and adaptations.

With this month's release, about 230 new tests have been added. The remarkable thing here, however, is not just the number of tests alone. The newly added tests relate to one of the most complex areas of Goobi workflow, in that we have now also achieved significantly better test coverage in the metadata editor.

Code quality

In addition to increasing test coverage by means of automated tests, we have been working intensively on the code quality of Goobi workflow. To this end, we have now registered Goobi workflow within Sonarcloud, so that we automatically receive code analysis results there and can make successive improvements to the software's source code.

There you can see, for example, how we achieved a level A rating for all important areas of quality analysis this month:

Here you can also see a comparison between before and after:

If you want to take a detailed look at the quality of the Goobi workflow source code, you can take a look here:

Plugins

Extension of the URN plugin for special construction

Not so long ago we reported here about a new plugin for registering URNs that uses the REST API of the German National Library. This plugin has now already gone into regular operation with the first Goobi users. In this context, we have then added an additional functionality: the construction of URNs can now use the timestamp of the registration if desired.

Previous use with a counter:

New possibility with the use of a timestamp:

We have also taken the opportunity to make the log messages in the operation log a little clearer again, so that they all receive a prefix and are thus clearly distinguishable from other log messages, as in this example:

And of course we have also expanded the documentary and added the new possibilities:

Here is also the link to the source code and the compiled version of the plugin on GitHub:

Deleting multiple images in the Image-QA plugin

We have found a malfunction in the popular Image-QA plugin. There, it is possible to delete images, provided that this has been activated. Apparently an error had crept in here, which caused that when selecting several images for deletion, not always the selected images were deleted. This has now been corrected.

More information about the plugin can be found here in the documentation:

And if you don't use the plugin yet, you should definitely try it. You can get it here:

Export-Package plug-in equipped with ZIP file functionality

With the Export-Package plug-in, a plug-in is available that allows almost all conceivable exports of data to third-party systems. It is possible to control exactly which data should be included in the export, where the data should be delivered and much more. Even a conversion of the METS files into third formats can be done by applying an xslt transformation. What was not yet possible, however, was that the exported result should not only be exported as a directory but also zipped afterwards. We have only developed this function and added it to the documentation accordingly.

Here you can find the detailed documentation of the plug-in:

And as usual, the source code of the plugin is here, together with the compiled version of the plugin:

New plugin for mass import from Excel files with EAD enrichment

We have implmeented and published a new plugin for the mass import of Excel files. The special feature here is that several structural elements are generated from each Excel file that is imported, one element per line. In addition, an existing EAD tree can be enriched with further nodes, so that an extensive inventory is created.

The functionality of the plugin can be viewed in detail here in the documentation:

And here is the source code of the plugin:

Version number

The current version number of Goobi workflow with this release is: 22.08. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Opac Plugin for Kalliope documented

The plugin has been around for a long time. But somehow we haven't found the time to document the plugin yet, and so one could almost think that Goobi has no connection to Kalliope. But this is not true, because this documentation proves that such a plugin already exists:

If you look closely at the source code published on GitHub, you can even see that the plugin has been around since November 2014. Who would have thought that?

Opac plugin for Ariadne extended

We already implemented a plugin for the connection to Ariadne some time ago. This has now been extended to allow more flexible control of the publication type. We also took the opportunity to document the plugin in detail and publish it on GitHub.

Here you can find the detailed documentation for the plugin:

And here we have published the plugin on GitHub:

New more flexible DOI plugin

About 1.5 years ago we already published a plugin that allowed the registration of DOIs. However, we were not entirely happy with the implementation. Especially if you take a close look at what can be reported in a separate metadata schema when registering persistent identifiers with DataCite, you realise that the previous solution had narrow limits there. For this reason, we have now implemented a second DOI plug-in that is much more flexible and could generate almost all conceivable data formats on the basis of XSLT.

Here we have documented the plugin in detail:

The source code for the plugin has been published here on GitHub:

Converting the inline help

Goobi worklow has had inline help for some time. This allows help texts to be displayed, especially for various forms, to support the input. Unfortunately, the problem was that the page was always reloaded when this integrated help was switched, so that values already entered or changed in the form were reset if they had not yet been saved.

There has now been a change in how the help display is activated. From now on, it loads solely via JavaScript and can thus be displayed without reloading the page. This avoids the problem of unsaved content being accidentally reset.

(Further) development of a plug-in for flexible export

Some years ago, we developed an export plugin that allows a very flexible export and where, among other things, it was possible to control which image directories should all be taken into account for the export. We have now massively extended this plugin and documented it in an exemplary manner. It now also allows, among other things, that within an export call, depending on an existing metadata, an export can take place in several Goobi viewers and, for example, different image directories can be used and different handling of OCR data should take place.

More information can be found here in the new documentation:

And here we have published the new plugin on GitHub:

Extension of the Goobi manual for configuration files.

Basically, the whole month was strongly focused on documentation. In addition to the newly created documentation for new and existing plugins, a lot of new things have been added, especially in the area of configuration files.

The following Goobi workflow configuration files have been newly documented or massively revised in their existing documentation:

goobi_metadataDisplayRules.xml

This configuration file is decisive for how the input fields should behave within the metadata editor:

goobi_rest.xml

This configuration file defines which areas of the REST API external services may access and how they must authenticate themselves:

goobi_opac.xml

This configuration file specifies the catalogues from which Goobi workflow should be able to import data:

goobi_opacUmlaut.txt

This configuration file controls how to deal with umlauts returned by the catalogue query:

goobi_projects.xml

This extensive configuration file allows you to influence exactly how the Goobi creation mask should be structured.:

Correction for GoobiScript when executing multiple setTaskProperty calls.

There were some special exceptions where the execution of multiple GoobiScripts only executed the last of the commands sent at the same time. Fortunately, this has now been found and fixed, so that even such similar commands can now be passed to Goobi and executed correctly..

Session validity period

For many years, Goobi workflow has offered the possibility for users to set an individual session timeout. Only very few users made use of this option and only because the default value was set to 30 minutes, which was too short. The new default value is now 4 hours.

Processing and provision of e-mandatory publications - Work in Progress

Together with the Zentral- und Landesbibliothek Berlin (ZLB) we are currently developing a completely new functionality for the processing and provision of e-mandatory publications. Overall, this is quite an athletic undertaking with, in part, enormous complexity. For this reason, the development of the functionality will take a few more weeks. However, in order not to keep the world in suspense forever, we would like to provide a small insight into the current status quo of the developments by showing various screenshots of the (still) only German-language interface:

On a specially adapted interface in the corporate design, the deliverers can log in or register in Goobi workflow.

Delivering institutions create their own user accounts in an interface that is as intuitive to use as possible.

After the ZLB has activated newly registered users for submission, they can add further data on the submitting institution.

After complete registration of the delivering institution, various actions are possible, including the delivery of independent works.

Within the user interface, it is possible at any time to have help texts displayed for input via an embedded help function..

Within the regular Goobi workkflow interface, a special administration plug-in ensures that the ZLB has an overview of all delivering institutions that are available as clients in Goobi.

The associated users of a delivering institution can also be managed via the administrative areas.

Among other things, the privacy policy, which the submitters accept during registration, can also be edited in this area.

All in all, a lot has already been done here. However, the work is still in full swing and will take some time. Some functions and interfaces will probably change significantly in the coming weeks. But as a first glimpse, the screenshots here already show well what will be available in the coming weeks and months as additional functionality for all interested parties as an open source solution.

Version number

The current version number of Goobi workflow with this release is: 22.06. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Coming soon

• GoobiScript changes

• Documentation of step details

Core

GoobiScript Templates

It often happens that several GoobiScript calls have to be made in succession. Often the calls are repeated because, for example, in a certain project under certain conditions it is necessary to insert a certain step again and again.

There is now a new mechanism to make these tasks more convenient. In the "Administration" section, the new GoobiScript Templates section has been introduced. There, multiple calls can be elegantly concatenated and saved as a template. Each template has a mandatory name and an optional description.

In the list of GoobiScript templates, the self-defined templates are displayed as badges with their own colour. The description is displayed when the mouse is moved over the badge. The same possibilities for dedicated rights management apply to the templates as to the built-in GoobiScript calls.

Properties during process creation

When creating a process, the process properties can now also be managed via the central configuration file goobi_processProperties.xml. For this purpose, the new XML element was added to the configuration file. The attributes are access="" and template="". The values of the access attribute are analogous to the other options already available in the file. The template attribute controls which production template the setting applies to. The XML element is repeatable to be able to configure different production templates. It is possible to define a * in the template attribute so that the setting applies to all production templates.

With this change, it is possible for the first time to visually separate the metadata that is imported from the catalogue from the properties that are to be collected when creating an operation and are therefore completely different in terms of content. Until now, both were displayed mixed together in the upper area with the heading "Import metadata".

Here is an example of a configuration and a screenshot of how it had to be done and looked in goobi_projects.xml:

The same configuration can now be mapped via goobi_processProperties.xml. A separate box is then displayed for this purpose in which the settings can be made. This makes it possible to realise a more clearly structured and thus clearer interface.

Snippets

• The newspaper export for the DDB now writes extended information to meet DFG viewer display requirements.

Code analysis

Version number

The current version number of Goobi workflow with this release is: 23.07.2. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Flip functionality in the QA plugin

Within the plugin for the quality control of files, it has already been possible for some time that specific individual files can also be changed, provided that this has been activated in the configuration. For example, images can be permanently rotated or even deleted. A new feature is that images can now also be mirrored. This is possible both horizontally and vertically.

To use this functionality, the configuration must be adapted according to the specifications. As before, the details can be found in the plugin documentation:

Clearer reporting of errors when extracting PDF files

If PDF files are to be used in the workflow, routines are available that automatically extract the contents of the PDF files and convert them to images. In the past, however, there was occasionally a display of irritating messages in the case of errors that occurred. This has been changed accordingly and is now much more meaningful and helpful for users.

Sorting of users now more intuitive

The sorting of the display of users within Goobi workflow was not quite intuitive until now. This has now been fixed and the sorting is more intuitive without the upper and lower case of names having unexpected side effects.

New GoobiScript for cloning processes

Once again we have a new GoobiScript. This time we needed a functionality that allows us to completely duplicate existing processes in Goobi workflow together with their associated data. The title of the process to be used can be freely defined using variables. It is also possible to specify whether only the data from the database or also all internal directories are to be duplicated.

This is what the new GoobiScript looks like:

And as before, the full documentation for GoobiScript can be found at this address:

Execution of a plugin from the transaction details now writes a message to the transaction log.

Until now, it was the case that the call of a plugin from the transaction details was not included in the transaction log. For better traceability, however, this would be practical from time to time. For this reason, we have now changed the logic of the execution so that such executions are also included in the operation log.

Ruleset file now dropdown selectable

For the configuration of rulesets within the administrative interface of Goobi workflow, the file names of the rulesets in the central ruleset directory /opt/digiverso/goobi/rulesets/ had to be specified until now. There has now been a small change here. Instead of having to enter the file name manually, it can now be selected from a drop-down list. This avoids possible typing errors when specifying rulesets for the future.

Breadcrumbs corrected after creating a process

When you have created and saved a new process within Goobi workflow, it is possible to go directly to the newly created process from there by clicking on the link Open the created process. The breadcrumb displayed there for navigating to the complete list of all processes used to occasionally lead to an empty hit list. This has now been corrected so that the navigation works correctly and lists all processes at this point.

Rights management: Bug fixed

Goobi workflow allows individual plugins to define their own rights, which a user must have in order to run the plugin. Accordingly, it is possible to configure such flexible rights in the user administration of Goobi worklow. However, until recently it was also possible to add empty rights there, i.e. without specifying a right. This has now been corrected so that such an error entry is no longer possible.

Ruleset file can be viewed in the activity details

In the past, when working with several rule sets, it was often necessary to check again in the area Administration - Rule sets which rule set file it actually was (e.g. for the rule set Standard). In order to make this search easier, we have now implemented a small functionality that displays this file name directly within the transaction details when you hold the mouse pointer over the rule set name.

By the way, we have already implemented the same functionality in various other areas in the past. For example, in the properties:

And there is also this hidden function in the metadata:

New backup system for meta.xml and meta_anchor.xml

The old backup system in Goobi workflow used to work only with a counter at the end of the file for the names of the backups. The file with the highest counter was the oldest backup:

This system had problems, especially in hierarchical file systems, because rarely used files are stored in a slower storage area. However, since the older backups were always renamed with the previous backup system, this sometimes resulted in very long storage times. Therefore, we have revised the mechanism of naming these backup files and now use a time stamp to name the backup files. The naming scheme is as follows:

Here, ii is the milliseconds in the second. An example of a new backup file looks like this:

Backup rotation: Only files managed by Goobi are treated

With the old backup solution in Goobi workflow, files not managed by Goobi were included in the rotation in the past (for example meta.xml.MYBACKUP). This resulted in manually created backups being rotated away or the manually created file not being saved to disk. As of this release, Goobi only considers files meta.xml.1, meta.xml.2, meta.xml.3 etc. and backups created according to the new scheme for rotation. Manually created backups, on the other hand, remain unchanged.

Better metadata collection for image areas

The fact that metadata can now also be recorded for page areas in the metadata editor has been possible for about 1-2 years. What became apparent during daily use, however, was that better usability was still needed to add several page areas to a structural element or to adjust the size of such page areas or even to delete them. We have now implemented some improvements here.

Für ein Strukturelement können im Bereich zum Ändern von Strukturelementen nun weitere Seitenbereiche ergänzt werden:

Und hier lassen sich bereits angelegte Seitenbereiche auch wieder einfach entfernen.

Excel import can now fail in a controlled way with missing images

Until now, the Excel Import plugin ignored missing image folders and continued with the next process. Our concern, however, was that in such a case of use, there should be transparent communication with the user so that he or she would also be better informed of the missing images. With the extension of the plugin, this has now become possible. The following excerpt from the configuration illustrates how such situations can be handled from now on:

Improvements when updating the database schema

After an update, Goobi workflow checks whether the database schema is still up-to-date. If it is not up to date, an update is usually performed automatically, bringing the version of the schema to the current version. In some cases, however, something went wrong with this update and the version number in the database was increased anyway. This resulted in an outdated schema with the current version number saved. However, the error handling has now been improved so that the current version is no longer written to the database after an error. This prevents possible incompatibilities in the future.

GoobiScript executeStepAndUpdateStatus now also closes the step for step plugins.

There was a misbehaviour in the GoobiScript executeStepAndUpdateStatus that prevented steps where a plugin is configured from closing correctly. This behaviour has now been corrected so that steps are now reliably closed when called by this GoobiScript.

Determine number of metadata and images: Order of tables changed

Previously, when clicking on the Get number of metadata and images link below the table of processes in Goobi workflow, the table with the results for each process was displayed first. However, as most users are more interested in the table with the summary, we decided to swap the order of the tables so that the summary now appears first without having to scroll through all the processes first.

Version number

The current version number of Goobi workflow with this release is: 22.02. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Step plugin for the generation of run sheets

We have developed a new plugin that automates the generation of dockets during the workflow. Up to now, it was already possible to download routing slips immediately after creating tasks or with a plugin within tasks. What is new is that routing slips can be created as PDF or image files with a new plugin and that they can be saved automatically in the desired resolution within a configurable directory.

More details about the plugin can be found in the detailed documentation here:

The source code of the plugin can be found published on GitHub:

Image areas can be captured better

It was not ensured in all cases that the image area capture in the metadata editor was only displayed if the associated rule sets also allowed this. We tested this again intensively and also made changes to the logic in order to avoid errors here. If we have thought of all areas, then no more difficulties should be expected here for the creation of new image areas, the addition of further image areas and the deletion of image areas for structural elements.

To ensure that we nevertheless take this well into account for updates, the update instructions have also been updated in this regard. There is a corresponding reference to the adjustments to be made within the rule sets:

Extension of the editors for configuration files and rule sets

We had not expected the two still quite new plugins for editing configuration files and rulesets to become so popular so quickly. In the course of their rapid spread, we soon realised that we still had to optimise a few things. Among other things, we noticed that the naming of the backup files should be changed again, that it would make sense to check and visualise the write permissions for the backup files and the files to be edited, and that we should change the way we display the success or error messages again.

So all in all, both editors should now be a little bit more practical than they already were. :)

Here again are the links to the updated documentation:

Icon for holdings in Archive Management

It is only a small thing, but in the end it bothered us ourselves so much that we are now happy about the change: The top node of each inventory within the Archive Management plugin was basically visualised incorrectly with a file icon. To avoid misunderstandings, this top node is now adequately labelled and thus self-explanatory for all users.

Automatic generation of preview images

For a workflow step, it is now possible to configure that preview images are to be generated for display in the metadata editor or image QA plug-in. The generation of these preview images takes place when the work step is to be opened. If the internal message queue is active, the processing takes place via it, otherwise the images are generated directly. Only after the generation of these preview images has been successfully completed is the work step released for users to accept. Automatic tasks, on the other hand, are closed directly after the preview images have been generated.

An example of a configuration for the generation of such preview images looks like this:

The preview images are thus created in the respective directories as configured.

Password length controllable via configuration

To ensure that users use secure passwords, it is now possible to control the minimum length that passwords must have when they are reassigned, when they are reset or when a user wants to change the password himself.

In the configuration file goobi_config.properties the following optional value for the minimum length can be specified from now on:

This configuration switch and many others are documented in the Goobi workflow manual here:

New plugin for metadata enrichment from an Excel file

There is a new plugin for enriching existing METS files on the basis of existing Excel files. Metadata and persons are read from a configurable file and transferred to the metadata of the process.

More details on the plugin and its configuration can be found here:

The source code of the plugin has been published here:

New plugin for data transfer from a directory

Another new plugin has been developed to read existing data from a directory and transfer it to METS. Metadata, structural data and pagination information are transferred from the directories. More details can be found in the documentation:

And this plugin is also published on GitHub:

New plug-in for manipulations within METS files

For a specific use case, we needed a way to automatically determine and interpret individual metadata within existing METS files in order to subsequently generate independent metadata within the METS file. The existing use case is currently still very much tailored to the needs of the Kiel City Archive and therefore may not be reusable unchanged for other Goobi users. However, the plugin offers a good basis for implementing similar specifications on this basis in the future.

The detailed documentation for this new plugin can be found here:

The source code of the plugin has been published at this address on GitHub:

Selection of a bulk import plugin simplified

In most use cases, there are not multiple bulk import plugins installed within Goobi. For this reason, we have made a small change to simplify the selection of plugins in the user interface. If there is only one plugin to choose from, it will now always be selected immediately. This is not only logical and more convenient, but also saves unnecessary mouse clicks.

Extension of the plugin for editing metadata

The existing plugin for editing metadata has been extended to add and delete metadata in the same way as in the metadata editor. This makes the operating logic more comprehensible for the user and the plugin allows additional application scenarios. The documentation of the plugin can still be found at this address:

Archive management: Naming uploaded EAD files

With the possibility of directly uploading EAD files within the archive management plugin, some user scenarios were not initially considered. One such scenario is, for example, that the uploaded files may have unfavourable file names or the file extension may be missing. Such cases are now intercepted and spaces are also replaced by underscores. In this way, naming problems of EAD files should be a thing of the past. However, if you still encounter problems with future file names for EAD files, we would be happy to receive feedback on this so that we can make adjustments again.

Errors in the GoobiScript metadataAdd are logged better

When executing the GoobiScript metadataAdd, logging in case of errors has not been exemplary so far. Specifically, we missed the display of information within the process log there. This has been corrected.

Revision of the SRU client

It is probably more exciting for technicians than for others: We have completely revised the implementation of the SRU client, as the programme libraries used there have reached end-of-life. With a new implementation, which is also used in many OPAC plug-ins, for example, we are now back to the state of the art.

Logging within the mass upload plugin

We noticed that the logging within the mass upload plugin was clearly too granular and thus no longer really helpful for administrators. We have made changes here so that logging now takes place to the extent that it can also provide relevant information about what happened during the upload in cases where this is necessary.

Logging more helpful for logging out

More Goobi users always bring more potentially different use cases and other infrastructures. And so we sometimes notice where we can or should improve something for ourselves. This is also the case here in the area of logging out users. Here we already wished for better logging when users have logged in. And now we have done the same for logging out. Every admin is happy about that, right?

Bugfix for the statistic "Memory increase over time

In the "Memory increase over time" statistics module, problems could occur with the display of large amounts of data. This was mainly due to the fact that the entered start time was not taken over and instead the first entry in the database was taken as the start date. This led to very large amounts of data (over 2500 data records), which the visualisation library used could no longer handle. The start date is now taken over correctly with the changes made and the display of not too large periods of time works again.

Version number

The current version number of Goobi workflow with this release is: 22.04. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Security vulnerability Log4j

Like many other well-known software solutions and websites, Goobi workflow was also affected by the security vulnerability within the Log4j programme library. At least theoretically. On the same day as the vulnerability became known, we updated the application, released a new version of Goobi workflow and updated about 200 servers. Security problems have therefore not occurred on any of the systems we look after.

During the rest of the month, we continued to work intensively on Log4j, logging in general and further updates, and we developed several automatic junit tests.

New GoobiScript: metadataBackup

Once again there is a new GoobiScript. It is called metadataBackup and allows you to easily create backups of the files meta.xml and meta_anchor.xml. These are the important internal METS files that belong to each process. Such a backup is particularly useful if automated changes are to be made to these files and a backup copy should therefore be created beforehand.

The call of this GoobiScript is simple without further parameters:

Excel exports in statistics corrected

For a short moment, some Goobi installations were affected by the fact that Excel files could no longer be generated from the statistics. The reason for this was the update of a programme library for Excel files and the associated change of the file format. We have corrected this with the update.

Corrected save interval error

If users adjusted the time period for the automatic saving of metadata, this caused loading problems when entering the metadata editor. This has been fixed.

Extension of the plugin for automatic deletion of content

The plugin for automatically deleting content from Goobi workflow has been extended. It is now possible to delete specific metadata and properties. This is useful, for example, if data is to be automatically anonymised after a set period of time.

The configuration options have been extended and documented for this use case:

The plugin itself for the installation as well as the source code can be found as usual on GitHub at the following URL:

Extension of the delay plugin for multiple configurations

The delay plugin for automatic pausing of workflows for a configurable time has been significantly extended. Instead of allowing only one central configuration as before, it is now possible to create a single configuration block per project or work step, as is the case with most other plugins. This means that the same plug-in can be used several times in different projects and workflow steps and can behave differently.

An exemplary configuration looks as follows:

And of course this plugin is also published on GitHub:

GoobiScript executeStepAndUpdateStatus adapted

The GoobiScript executeStepAndUpdateStatus ignored the configured queues when calling steps. This has now been fixed, so that the execution is regularly enqueued in the configured message queue and thus correctly prioritised.

Execution of scripts within the process details

The execution of scripts within the process details also previously ignored the execution within a queue. This has been corrected so that they are now also executed within the message queues and correctly prioritised.

Plugin overview now also shows content from the goobi/lib folder

The handy plugin overview with its version display for each individual plugin is a great help to Goobi administrators for maintenance. After all, it is easy to see whether all plugins are compatible with the Goobi version installed. Missing from this list, however, are those plugins that are stored in the directory goobi/lib. In order to be able to see them in the list and check their status, the overview page has been extended.

Accessibility: Dropdown buttons revised

Again this month, there have been some accessibility improvements. Among other things, all buttons that have a dropdown functionality have been adapted so that their state (open vs. closed) can be processed well by screen readers.

Version number

The current version number of Goobi workflow with this release is: 21.12. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Extension of the user settings for individual configuration of the display mode

Within the user settings, there is now the option in the General section for each user to individually define the display mode. You can choose between Normal display, Low vision mode and the WCAG compliant mode.

This option is of course in addition to the possibility for each user to customise the layout of Goobi themselves via their own CSS definitions.

Adjustments to the configuration editor

In the configuration file editor plugin, it is now possible to configure a list of directories in which files are to be searched for editing. The new configuration now looks as follows:

In this context, it was considered that the editor should not be used to change its own configuration file. This would otherwise have allowed authorised users to define additional paths for the editor in order to carry out more profound interventions in the system.

The documentation for the plugin can be found here:

Skip Link

As part of our ongoing improvement of the user interface to increase accessibility, we have now also taken a closer look at the central Skip Link. This is located at the top of all pages and allows users with keyboard control to navigate immediately to the content area of the page via the TAB key. So now there has been a small correction for the correct order of scrolling through the page elements.

Tasks from production templates no longer in the dashboard

Within the dashboard, which the majority of Goobi workflow users use, the most recently completed tasks are displayed. In the past, tasks from recently processed production templates were inadvertently listed there. A correction has now been made to avoid this display error.

Comments on individual images now possible

With the new developments, it is now possible to write comments on individual images. This comment function is integrated in several places in Goobi workflow: in the METS editor, in the ImageQA plugin and also in the LayoutWizzard. To use this new functionality, it must be activated within the central configuration file goobi_config.properties:

Thumbnail selection only affects pagination

Until now, clicking on a thumbnail in the metadata editor always set the current page in the pagination, even if you were in the structure data view, for example. This could inadvertently lead to operating errors if further pagination work was to be carried out later. By changing the operating logic, a click on the thumbnails now only changes the page selected in the pagination if you are actually in the pagination.

Error in the process log when Enter is pressed

There was an error in the transaction log when a user pressed Enter after entering a message. The display was changed there instead of sending the message. This error is now fixed.

Checking all buttons

All buttons within Goobi workflow have been intensively checked to ensure that they continue to function and are displayed correctly after the extensive accessibility changes. If we have overlooked a button or functionality, please let us know.

Invalid Goobiscript syntax now directly refuses to work

Up to now, the GoobiScript parser first displayed the confirmation prompt when invalid YAML syntax was used and then displayed an error at the very top of the page. This led to confusion among users because the previously sent GoobiScript was not executed as expected after all. For this reason, from now on the GoobiScript will be checked for validity directly after it has been sent and, in the event of an error, an error message will be displayed directly above the GoobiScript area.

GoobiScript "addUserGroup" can now be successfully called multiple times

There was a misbehaviour in the GoobiScript addUserGroup which, when called multiple times, resulted in only the last added user group being added to the desired task. This behaviour has now been fixed and the GoobiScript behaves correctly from now on.

No more leading or trailing spaces in vocabulary management

In the vocabulary management, leading and trailing spaces are now removed from all text fields (Input, Textara, HTML) to avoid such accidental entries and the potential problems resulting from them.

Metadata changes via GoobiScript using VariableReplacer

The GoobiScripts metadataChangeValue, metadataAdd, metadataDelete and metadataReplace now use the Variable Replacer for content changes, allowing very flexible use involving other metadata field contents or even information about the process to which the metadata belongs.

An example call looks like this:

As before, more information is available here on the Goobi workflow variable system:

Version number

The current version number of Goobi workflow with this release is: 22.01. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

November 2021

Better display in case of metadata that cannot be imported

When a process is created in Goobi workflow and data from a catalogue is to be imported for it, it can occasionally happen that it has not already been configured for all the metadata provided where within the internal METS file the metadata is to be stored. This was previously not transparent for the user, so we have made a change. Metadata that cannot be transferred are now clearly visualised in the creation process.

ALTO editor can be called from the thumbnail view

Since a few versions, Goobi workflow has offered the possibility to display and edit the results of a full-text recognition in ALTO format directly from the metadata editor. This functionality was previously only accessible from the single page display. With the changes made, however, this is also possible from the thumbnails and subsequently allows navigation between the images within the open editing screen.

ALTO Editor supports ALTO 4.0

The already mentioned editor for full texts in ALTO format now supports the ALTO files in version 4.0 and allows their display and editing. This means that it is now also possible, for example, to display and edit the full texts of Transkribus.

Various extensions to the archive management plugin

The archive management plugin is being used productively in more and more projects. In this context, new wishes, necessary adaptations or even visual changes are always needed, which we gradually implement. Recently, there have been several optical adjustments to the entry mask, which make it worthwhile to take a look at the functionality again.

As a reminder, here again is the detailed documentation of the plugin:

Accessibility as an ongoing topic

With some developments, even with a lot of professional experience, one is still surprised at the extent to which they can take on and what one would not have expected. The topic of accessibility is a good example of this. We have already invested countless hours in optimising accessibility for Goobi workflow and already meet common validators. And yet there are always new surprises, as the following example shows:

This screenshot shows what effects changes can have if, for example, you change links to buttons so that screen readers, among others, can also handle them better. Of course, we didn't leave it like this, but styled it nicely again. However, if you find something similar in another part of Goobi workflow, please let us know so that we can correct it.

Fine-tuning for accessibility

A few months after the actual accessibility work was completed, we noticed a few details that could be improved. These included the correct attributes in tables so that screen readers can interpret them correctly.

The compatibility with the common validators remains under constant observation and we try to maintain 100% compatibility. However, it is important to keep in mind that validators change over time in terms of requirements and therefore constant monitoring of compatibility remains necessary. So the topic as such is probably never really finished. At the moment, however, it looks to Goobi workflow as if we have done our homework well:

Mass Upload Plugin Customisation

The mass upload plugin is especially useful when many processes in Goobi worklow are to be automatically supplied with the corresponding image files. Experience has shown that this works quite well. However, the button for assigning the images to the identified processes allowed a double execution by repeatedly clicking on it. This then led to undefined behaviour, so this had to be changed. The corresponding button is therefore no longer displayed if it has already been clicked and images are being sorted into the processes.

By the way, we took the opportunity to make a small correction: If the parameter use-barcodes was set to true in the configuration and no barcode could be determined when uploading images, a meaningful error message is now displayed.

Correction to Excel import

This item comes from the category "Unexpected side effects": In fact, a faulty configuration of the Excel import plugin could lead to the dashboard and the listing of process templates in Goobi workflow no longer being displayed correctly. We have made corrections here so that everything looks good again.

Table layout in the user administration improved

Within the user administration, the display of the column with the buttons was too narrow, so that the buttons wrapped awkwardly. This has been remedied by making the column at least as wide as possible, so that it looks good even with many projects and user groups.

Deleted and anonymised users removed from tasks

When users are deleted in Goobi, the user's record is not actually deleted in the background, but completely anonymised. This is done to ensure that the data as a whole remains complete. To make it clear that such users cannot actually be active in the workflow, such deleted users are now also removed from the list of authorised users for a task, provided they were assigned there. This is primarily a visual correction, as the user accounts were already deactivated and accordingly can no longer log in.

Automatic release of almost all plugins as compiled files

The fact that Goobi worklow and almost all plug-ins are under an open source licence should already have become common knowledge. There are currently over 300 plug-ins for Goobi workflow and about 130 have already been published on GitHub at https://github.com/intranda. Many of the plug-ins already have quite detailed documentation at https://docs.goobi.io, which explains, among other things, how to install the plugins. Until now, however, you had to download and compile the plugins yourself as source code. This was cumbersome and very technical. Therefore we have changed this so that compiled versions of the plugins are now automatically stored on GitHub whenever we make a change to a plugin or release a new Goobi version. You can access these compiled plugins by clicking on Releases in the right-hand pane of the respective plugin.

Afterwards, one gets an insight into the downloadable files, which include the plugin itself as well as the possibly required configuration files. These files can now be downloaded and installed correctly according to the instructions, so that commissioning has been simplified considerably.

Version number

The current version number of Goobi workflow with this release is: 21.11. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

New GoobiScript for changing person roles

For some time now, there have been various GoobiScripts that can very conveniently make changes even to metadata in bulk. In the past, we have often focused on regular metadata. With a new GoobiScript we now also allow changes to persons. The reason for this is that we needed a convenient way to change the roles of persons in bulk, for example, to be able to make inventors instead of wrongly assigned person roles as authors in 100,000 transactions. In such a case, the call to the new GoobiScript would look like this:

As always, this and the many other GoobiScripts are described in detail in the online documentation.

Here is the link to further help:

New documentation for developers

It took a long time. But it has turned out really well: There is now very detailed documentation for future Goobi workflow developers. It shows in detail how to set up a development environment, which tools are needed for commissioning and which settings should be made here and there so that development runs smoothly.

In order to make it as easy as possible for future developers, we have also documented in the instructions how to set up a small, meaningful dataset very conveniently, so that development is also possible with actual meaningful data. For this, we benefited from our many experiences with Goobi-to-go.

So if you've always wanted to help develop Goobi, it's now easier than ever. Click here for the documentation:

Export button now in the "Actions" column of the workflow details

Goobi worklow has for some time allowed exports to take place within a workflow via plugins. This is sometimes necessary for individual exports to special systems with special format requirements. However, if several exports are to take place within a workflow, it was previously not possible for administrators to specifically execute a particular export of a workflow. However, the exports already functioned as desired when the workflow was run through.

To make it easier for Goobi administrators to selectively call up individual export steps manually, export buttons are now displayed within the process details directly next to the relevant tasks. This means that they can now be called up as desired.

Better logging when changing a step status via GoobiScript

We noticed that the logging within the process log was still in need of improvement when the status of tasks was changed via GoobiScript. We wanted both the old and the new status to be easily traceable in the process log. For this reason, we have now made a small adjustment that makes such status changes more comprehensible.

Naming of backup files in the editor for configuration files and rule sets

We had recently already changed the naming of the backup files for the METS files of the Goobi processes so that they are better traceable in the editor, do not rotate further and thus cause unnecessary write accesses to slow memory areas. In order to implement this naming scheme consistently, we are now also using this mechanism for the automatically generated backups of the files that are created within the still quite new editors for the configuration files and rulesets.

Here, too, the naming of the backup files is now such that the date and time of the backup is immediately apparent from the file name and is a good help if the worst comes to the worst.

WCAG: Changes here and there

We have already described several times that the topic of accessibility is not only of interest to us but also occupies us full-time. In this respect, it is only logical that we have again improved many small things in favour of accessibility this month. This mainly concerned the display of the process log and some adjustments for the display of the process list.

But it will certainly not be the last time that this item is listed here in the digests 😀

Implementation of an ARK registration

We have already integrated various mechanisms for persistent identifiers into Goobi workflow in the past. In addition to URNs, Handle and DOI have also found their way into the Goobi universe. Recently, support for ARK identifiers was added. This is another system that, in interaction with the Goobi viewer, allows permanent linking of works and also granular linking at page level. Unlike other systems, ARK identifiers do not require a licence fee and can be freely assigned independently. More information on ARK in general can be found here:

And here you can find the documentation for the plugin we developed:

The source code for the plugin is published on GitHub here:

Ruleset as new column for processes and process templates

Sometimes it is helpful to see at a glance which ruleset is used for a process template. For this reason, the ruleset can now be displayed within the listing of processes and process templates. This is what it looks like:

Plugin for sending mails

A new plugin for sending mails within a task has been developed. The list of recipients and the text can be configured individually for different steps. All fields from the variable-replacer are also available. This means that metadata or information about the task, step or project can also be accessed in order to use them within the mail.

The detailed documentation for the commissioning of the plugin can be found here:

And at this point the plugin can be found as source code and in compiled version:

Extension of the user settings for individual configuration of the display mode.

In the "General" area, there is now the option to individually define the display mode. Here you can choose between normal display, low vision mode and WCAG compatible mode.

Plugin for exporting newspapers to the DDB newspaper portal

We have developed a new plugin for exporting newspapers to the DDB newspaper portal. The plugin is used to create the specific METS structure required by the DDB for newspapers. A METS anchor file is created for the entire record of a newspaper. (https://wiki.deutsche-digitale-bibliothek.de/display/DFD/Gesamtaufnahme+Zeitung+1.0)

For each exported volume, another METS anchor file is created and linked within the overall record. The volume contains further structures for month and day. (https://wiki.deutsche-digitale-bibliothek.de/display/DFD/Jahrgang+Zeitung+1.0)

Each issue, on the other hand, is created as a single METS file and linked in the METS anchor file of the respective year. The issue may contain further structural data such as article descriptions or supplements. The digitised images are also referenced here. (https://wiki.deutsche-digitale-bibliothek.de/display/DFD/Ausgabe+Zeitung+1.0)

Detailed documentation for the plugin can be found here:

And at this point you will find both the source code of the plugin and the compiled version of the plugin:

Extensive changes in the internal project structure

The development of Goobi workflow started in 2004. In the meantime, many new functionalities have been added and the application has grown many times over. In order to maintain a good overview, but also to comply with established best practice for software development (Maven directory structure) and to make it even easier to keep the programme libraries used up to date in the future, we have massively revised the project structure. This should not result in any functional changes for the users. For us as developers, however, these are extensive interventions in everyday work and project organisation. If you notice any undesirable side effects when using Goobi workflow in the near future, which should certainly not be desired, please let us know.

Incidentally, these changes have also already shown some positive results. The updates made to various programme libraries as a result of the changeover have already led to a measurable increase in the security of potentially vulnerable code components of external libraries and a better ability to automate testing for possible future security vulnerabilities.

Display error in a dropdown in the own tasks

There was a small display error within the own tasks. There was an idiosyncratic line break with unusual character representation that did not belong there. We have corrected this. Nothing happened 😀

Version number

The current version number of Goobi workflow with this release is: 22.03. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Display of the number of images in tasks and activities

Within the user interface, it is now possible to display the number of images as a column. This can be done within the task list as well as in the own tasks. This looks like this, for example:

Installed plugins better manageable

We have slightly revised the incredibly practical display of installed plug-ins and their versions. All existing plugins are now listed correctly sorted and the paths to the directories are also listed.

Better sortability of tasks and assignments

In the list of tasks, we have made some improvements regarding sorting. The biggest noticeable difference will probably only be noticed by administrators:

All columns now sort lexicographically in ascending order when you activate the sorting of a column. Only the column with the ID behaves differently and sorts in descending order at the first click, in order to display the most recent transactions immediately at the top. Administrators will be pleased. :)

Project structure optimised

Goobi workflow has been using Maven as a build system for a number of years. Last month, the project structure within Maven was further consolidated so that there is a dedicated module for running automated tests and checks within the Continuous Integration infrastructure.

The previously existing modules to build a .war file for deploying the known application within a Tomcat and the module to build a .jar file have been cleaned up and are now only responsible for compiling the artefacts. A new addition is a CI module that bundles previously scattered steps in the lifecycle and also integrates new checks. These are in detail:

• Running the JUnit tests

• Checking the dependencies used for known security vulnerabilities (OWASP)

• Creating an analysis of how much code is covered by unit tests (Jacoco)

• Checking the source code for violations of selected checkstyle rules

• Performing a static code analysis using Sonar

At the same time, the execution of the unit tests was changed from a suite-based infrastructure to a class-based one. The number of regularly checked tests could be massively increased. In addition, the last code sections that required compiling with Java 8 and explicit compiler options were cleaned up.

The changed project infrastructure now allows for faster building of the individual modules and thus less overhead in development, as well as expanded and improved testing of daily work to increase code quality.

Anyone who would like to track our progress in terms of code quality is welcome to do so here at any time:

The diagrams still show that we have a long way to go. But we are on the way.

And internally, progress looks something like this at the moment:

So things are moving forward. We'll report on it here more often in the future.

Better log information about executed GoobiScript

In the past we often noticed that we would have liked to have more logging if a GoobiScript did not do what we had hoped. We have now addressed this issue and solved it in such a way that administrators can easily see in the log files what was called. It now looks something like this:

Extensions for the AEON plugin

Some time ago we had already reported that we were implementing an extensive interface to the software AEON. There have been numerous extensions here again. Among other things, the look of the interface was massively revised.

Properties can be individually adjusted or adopted for entire datasets or even for individual objects that are actually to be digitised, with deviations from the default values clearly visualised. In addition, it is now possible for processes whose digitisation workflow is already in progress to be stopped, unless a configurable status has already been reached.

So, all in all, great progress, but also further work in progress.

GoobiScripts for metadata manipulation extended

There are already a lot of GoobiScripts. But every now and then we notice that a certain functionality would be handy. This time we noticed that it would be quite practical if we could manipulate metadata not only in the logical areas of the METS files, but also within the metadata for the images. No sooner said than done. Besides the parameters work top child and any it is from now on possible for the GoobiScripts to understand and apply physical for manipulating the metadata. More details can be found here in the documentation:

Changes to the database driver

To make Goobi workflow better able to handle newer MySQL databases, we have replaced the driver for communicating with the database. From now on we use the following driver:

It is now possible to run Goobi workflow with MySQL versions 5.7 and 8. However, the earlier versions are no longer supported. This must be taken into account in future updates.

This is already mentioned in the update instructions:

Extension of the Goobi manual

As you can see from our digests and plugins, the documentation of our developments has been very important to us for several years. Currently, we have also turned our attention to the configuration files. This area is still a work-in-progress. But if you want to take a look, you will find a few good examples here:

There will be a lot more happening here in the next few weeks.

Better display of information in the process log

In the case of failed script calls, the process log was not really very helpful until now. This has been adjusted so that incorrect calls now appear with a clear error message in the process log.

New user: Password was not a marked mandatory field

Incorrectly, when creating new users, it was not displayed that a password for a new account was a mandatory field. This has been corrected.

Version number

The current version number of Goobi workflow with this release is: 22.05. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Core

Generation of URLs for the SRU Interface and the IIIF Manifest

In the project settings there are now two additional fields in the tab METS Parameter to record the URL to the SRU interface of the Goobi viewer and the URL of the IIIF Manifest for the complete work.

Example entries could look as follows:

The result within a METS file created in this way is then accordingly as follows:

Adjustments for the image comments

Not too long ago we introduced the new image comments here. With their first use at various institutions, however, we noticed that the internal storage location for the data was not ideal. For this reason, we have made a change for this. Those who have already used this functionality should therefore follow our update instructions provided here for the next update:

GoobiScript metadataChangeType fixed

In the GoobiScript metadataChangeType the parameter any did not work as expected. This has been corrected and now allows metadata to be changed at all levels of a METS file as desired.

For the sake of completeness, here is the link to the detailed documentation of GoobiScript:

Improved logging during application stop

Administrators love logging. It helps to better understand why a problem occurred in case of an error and what can be considered as the trigger for a failure. For reliable operation of productive applications, meaningful logging is not only helpful, but often simply mandatory. It is all the better if the developers of a software work closely together with the administrators who actually run the software.

The result of such cooperation in the context of Goobi is that we are constantly trying to improve the logging of Goobi workflow. This month we took a closer look at the logging of the application stop and made some improvements. After all, sometimes too much logging is not right.

Plugins

Processing newspapers

We had already reported on this at the Goobi Days. In recent months, however, our developments for processing newspapers have been used more and more frequently. That is why we would like to explicitly address this topic again together with the latest developments:

For some time now, it has been possible for Goobi to recognise newspaper editions semi-automatically and to index them with metadata. To this end, the workflow first automatically recognises newspaper editions on the basis of trained models. We have recently expanded this considerably and fed it with more ground truth data. Thus, the recognition accuracy of newspaper editions has improved many times over.

After this automatic recognition, a plugin is available in Goobi worklow for enriching the newspaper editions with edition-related metadata. This makes it relatively easy to record issue numbers, dates and publication days, including morning and evening editions, without having to do this individually for each issue. Instead, a semi-automatic mechanism takes effect here, which counts up the subsequent issues accordingly with a few entries from a user, enriches them with dates and also names them.

In addition, supplements can also be created quickly here, so that a comprehensive METS file is subsequently generated, which has standardised date information and can be searched in the Goobi viewer, e.g. via a calendar entry.

To meet the requirements of the German Digital Library and the DFG Viewer, by the way, an additional export plugin was recently added that prepares and provides the data according to the specifications.

We have worked a lot on the individual components described here in the last few months and have repeatedly optimised their operation. Some things will continue to happen here in the coming weeks. We would be happy to present this in detail in a workshop at the upcoming Goobi Days.

Mass imports with archive stock enrichment

Up to now, we had not even mentioned the various newer import plugins that we had recently implemented for some specific mass imports. For these plugins, there is no detailed stand-alone documentation yet.

However, it is important for us to point out that there are several new mass imports that not only create processes in Goobi workflow, but also increasingly create or enrich entire archive trees with tectonics.

Depending on the project, structured METS files can also be created here, as with other importers for Goobi workflow, where structural elements and associated metadata are imported together with authority data for the associated image files.

Better full text extraction from PDF files

The step plugin intranda_step_pdf-extraction allows full texts to be extracted from PDF files and saved as ALTO files. However, we recently noticed an error in the processing, which was responsible for inappropriate gaps appearing within some words, among other things due to an existing hyphenation in the words.

We were now able to find and fix the cause of this problem. By the way, the plugin was already documented here:

We have published the plugin at the following URL, where the corrected version can now also be downloaded:

Documentaries

A lot has happened again in the area of our documentation:

Inline documentation for GoobiScript

In the GoobiScript addToProcessLog the included explanations were not ideal. There was a small correction here, which now looks like this:

Small change in the update instructions

In order to make the update instructions for Goobi workflow more intuitive, they are now no longer structured by date, but by the respective published version. This change only applies to all future notes that are included there and not retroactively. It should hopefully make it a little clearer for administrators in the future.

Configuration files

We have again put a considerable amount of energy into documenting more configuration files. Almost all configuration files are now documented there in an exemplary manner. Some remaining work is still to be done. But if you want to have a look, here are some sample links to new documentation:

Configuration for dockets:

Configuration of the queues:

Configuration for properties:

Configuration for translations:

Configuration for the Web Api:

Configuration for the authority data:

Configuration for the field display in the metadata editor:

Version number

The current version number of Goobi workflow with this release is: 22.07. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

Coming soon

• Extension of the plugin for issue recognition of newspapers

• Support of S3 as storage in further plug-ins

• Enhanced accessibility for tables

• AEON plugin development

• Documentation and code releases

Extension of the display of available user rights

Already since a major change a few years ago, plugins can bring their own user rights. These can then be used in addition to the rights that Goobi workflow itself already brings. This is usually done by opening the plug-in to find out the name of the required right.

Then, within the user groups, you can enter the required right in the lower right area and thus also assign it to the user group.

In the case of very complex rights, of which individual plug-ins can also have several, the determination of the required rights is sometimes too cumbersome. For this reason, Goobi workflow has now been extended so that within the central Goobi configuration file all those rights can be made known that Goobi should list within the list of available rights. In this way, the rights of plug-ins are just as selectable and clickable as those already provided by Goobi itself.

The configuration of these additional rights is done within the configuration file goobi_config.properties. It looks as follows as an example:

userRight=RightA
userRight=RightB
userRight=RightC

If the rights entered in this way are also to be displayed with suitable translations, these must be entered accordingly within the local messages_XY.properties files. This must be done as an example:

rights_RightA=Right A for the user group
rights_RightB=Right B for the user group
rights_RightC=Right C for the user group

New plugin to bring batch operations to a common status

As part of a project where Goobi is to notify another technical system (AEON) via an API when all operations in a batch have reached a certain status, a new plugin has been implemented. This is a step plugin that stops further execution of the workflow for the respective operation if there are other operations belonging to the same batch that have not yet reached the same status within the workflow. The last process that reaches the desired status triggers all processes to switch to the next process step together and thus continue to run through the workflow. This plug-in therefore serves to bring all processes back to the same status in the workflow. If desired, the plugin can also send a request to AEON via REST to inform about the change of status.

Detailed documentation for the plugin can be found here:

And this is where you will find both the source code of the plugin and the compiled version of the plugin:

Catalogue-Request-Plugin and Catalogue-Poller extended

The two plug-ins that allow simple or repeated catalogue queries during the already running workflow have been extended to work with the new functions of the JSON-OPAC plug-in. Since multiple fields can be used there for a single query from a catalogue, i.e. to import a result with multiple search parameters, the configuration of both plugins has had to change. For updates of Goobi or the plugins, the configuration files have to be considered accordingly. Details on this can be found both within the documentation of the plugins here:

... and here:

And of course the necessary adjustments are also listed in detail within the update instructions:

New plugin for editing configuration files

In the past, it was difficult to make small adjustments to Goobi when there was no direct access to the server. With the new plugin for editing configuration files directly from the Goobi user interface, this is now different. If a user has the necessary rights in the system, it is now possible to change configurations directly from Goobi.

In the background, the plugin creates backups in a definable number, so that an administrator can also restore an earlier version on the server. In addition, the plug-in also offers the option of integrated help. Here, you can determine yourself whether your own help texts should be displayed for editing configuration files. These must then be stored accordingly within the so-called message files and can even contain formatting information.

More details about the plugin can be found in the very detailed documentation here:

The source code and a compiled version of the plugin can be found here as usual:

Logging and login

When users logged in to Goobi workflow, it was difficult for us as developers to understand what was causing the technical problems. For this reason, we have taken the time to look into this issue and have thoroughly revised the writing of log files, especially with regard to logging in. The balancing act here is, of course, that we are allowed to log as little data as possible in view of the GDPR We have now thought of all these issues and adapted the logging accordingly. So we can work well with it in the future. And if someone wants to see it in action, they can simply try the following command on the server side:

cat goobi.log | grep "LOGIN PROCESS"

Institutions are only displayed to a limited extent

About two years ago we massively extended Goobi workflow to allow better multi-client capability. This is based on the fact that several institutions/clients can now be defined in one Goobi instance.

However, if several institutions were not defined in a Goobi instance, a new selection field was unnecessarily available in many areas, in which usually only one institution could be selected, because the use case of several institutions in one instance occurs relatively rarely. Therefore, we have now made changes in various places in order to only show this display of institutions if there really are several institutions. Otherwise, both the selection field and the table column are basically hidden. This way, the interface remains tidy and clear and does not display unnecessary information.

Project names more prominent in editing

We are gradually changing some visual subtleties so that the operation of Goobi becomes even more intuitive. One such refinement is, for example, that when editing data sets, we now list the name of the respective data set much more prominently in the title of the page. The screenshot shown here illustrates this quite well, because it displays the name of the edited project in large letters at the top, even if you are on a tab within the editing form. This makes it clear at all times which project you are currently making changes to, for example.

New plugin for editing rule sets

In order to enable the editing of ruleset files no longer only via server access, we have developed a new plugin. It allows editing of rule sets as xml files directly from the Goobi interface.

The loss of changes made within the files is avoided by the fact that a change to other rule sets always also checks whether there may be unsaved data.

In case something goes wrong, the plugin automatically creates backups in the background in a definable number, so that even accidental changes can be undone by an administrator on the server.

Please note: Even with this plugin, one should know what to configure here and how. The facilitated access to the rule sets can lead to serious data loss of previously recorded metadata if used incorrectly.

A very detailed documentation for the plugin can be found at the following address:

The source code of the plugin together with a compiled version of the plugin can be found on GitHub:

Better feedback when uploading files

When uploading files with the file upload plugin, misleading error messages came up in the past when a validation did not want to accept the uploaded files. Until now, the same not really helpful message came up that no objects could be found. The actual reason for the message in such cases was that files were found, but they did not correspond to the validation rule. We have now revised this and display clearer messages for such use cases.

Version number

The current version number of Goobi workflow with this release is: 21.10. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>21.10</version>
</dependency>

Coming soon

• Extension of the Archive Management Plugin

• Extended Java 11 support

• Adaptation of the export options

• Extension of the handling of museum data from LIDO

New functionality for full text editing in the metadata editor as ALTO files

For some time now, the Goobi workflow metadata editor has allowed OCR results to be viewed in text format. However, if you then discovered a recognition error, you were left with only three options:

1. use the transcription plugin in Goobi workflow and export text instead of ALTO-XML, which however prevents the marking of search hits on an image in Goobi viewer.

2. edit the ALTO file with a separate text editor in the file system. This is very time-consuming and hardly feasible for non-professionals.

3. do not correct the recognition error and try to sleep peacefully anyway.

Since this Goobi workflow version there is a new possibility:

You can use the new integrated ALTO editor within the metadata editor. The coordinates of the words are thus preserved and the full text can be edited almost like regular continuous text.

Highlighting the lines and words allows it to be easily seen where on the page each word is printed.

The documentation for the new functionality can also already be found in the Goobi workflow manual available online:

https://github.com/intranda/goobi-workflow/commit/50302f4464c417f090593a91151eb3c127643223 https://github.com/intranda/goobi-workflow/commit/4ebd2aa551680117bd8ef6f224f030a31caaef3c https://github.com/intranda/goobi-workflow/commit/04b4cddefcebb8d8675c116946bff3c1df4a9fda https://github.com/intranda/goobi-workflow/commit/293e19fc4cdc3289d4f6049868578bc7d4e21d61 https://github.com/intranda/goobi-workflow/commit/c0dcfcb477ef0c3266db1d18c7ff5da495f1835b https://github.com/intranda/goobi-workflow/commit/ef8609c9de7aeee3e00d66ff62a7545d1f88c00e https://github.com/intranda/goobi-workflow/commit/c619c43bdbd02763391e253056e44b3fc4953c2f https://github.com/intranda/goobi-workflow/commit/0a4350efff933aed8a6196074f40329591b19efa https://github.com/intranda/goobi-workflow/commit/94db988c85e3861fcad3c946917e353a697a1824 https://github.com/intranda/goobi-workflow/commit/e82ee6d1a2d8544a692141affda6fbffce51e31b

Layout adjustments in the metadata editor and faster loading times

When an action is triggered in the metadata editor, the entire page is rarely reloaded so that the workflow is not interrupted and the scroll position of the individual containers can be maintained. When refreshing the page in this way, the developer can decide exactly which parts of the page should be refreshed.

In the past, some actions also updated parts of the page where it was not necessary. As a result, some actions, such as setting the representative, took an unusually long time. We have completely checked the metadata editor for this problem and optimised the updating for all actions.

https://github.com/intranda/goobi-workflow/commit/c8c4bb7bf26e3127795e5efef016f6c7bd88a36b https://github.com/intranda/goobi-workflow/commit/8d8fc0ba9de8344d1f8d7fe7ed1a4f6f99e2dba2 https://github.com/intranda/goobi-workflow/commit/e583242c440624e4b6ee8c63691c8ed7bfd7e993 https://github.com/intranda/goobi-workflow/commit/f472088c86c579faffabd33800cda5960d9227c2 https://github.com/intranda/goobi-workflow/commit/5c43b1e89e32ccf9927c2770354d78950213a26b https://github.com/intranda/goobi-workflow/commit/c390ceb75df8c4219711c9ef1fa9d0d754dd5844 https://github.com/intranda/goobi-workflow/commit/5550ce2393456cc214f1f21787fa49c91f841616 https://github.com/intranda/goobi-workflow/commit/27a0653758593f24d7e82ef645f239ed0805f8d2 https://github.com/intranda/goobi-workflow/commit/5cf7bd6e7b3176ae4692df5a25373f8474db49f6 https://github.com/intranda/goobi-workflow/commit/827aa52ff63663fa1aab7028b32786e4e5206976 https://github.com/intranda/goobi-workflow/commit/7cc545c634bd355d0d5b7ea66e39d5261930376c https://github.com/intranda/goobi-workflow/commit/020f270d011bb10d81ef23dddec9d095c9c2be57 https://github.com/intranda/goobi-workflow/commit/5ccfbf44e4c57302b834b8a04d7d0638ec229f05 https://github.com/intranda/goobi-workflow/commit/adf93e08dff00cf879fb4c4436cf935e156c6cf2 https://github.com/intranda/goobi-workflow/commit/fc516f07f9170849331be0271ae093b5e38f9cdb

Avoiding unwanted caching for layouts

In the past, it was often the case that after an update, the first time you entered the just freshly updated Goobi, the layout and/or some stylings were completely broken. This was not because the developers did a bad job, but because browsers like Chrome or Firefox cache files on the local device to save resources and bandwidth. The files are only re-downloaded from the server if they have changed. Therefore, it sometimes happened after an update that old files were mixed with new files and therefore the layout was broken. In this case, the page had to be completely reloaded (with the key combination Ctrl+Shift+R) to discard the cached files and reload them.

To prevent this from happening in the future, we now append a unique text associated with the version to the address of each file that could potentially be cached locally for each Goobi release. This way, the browser will always request a new file for new versions and the layout will look correct immediately after the update, without having to press secret key combinations.

https://github.com/intranda/goobi-workflow/commit/b597c0439d3f5d6004e82811929c6156100ce167

Better handling of processes without metadata

Goobi workflow is used in many different ways. For example, there are some institutions that are not interested in metadata in METS/MODS or LIDO format. For these institutions it has been noticed that script calls mandatorily require a meta.xml file in the task folder, otherwise they are aborted with an error. This has now been changed so that the meta.xml file is only read for script calls if metadata information is actually required within the script call.

https://github.com/intranda/goobi-workflow/commit/d7d7d2ca5fa60a59c2ef3a996eaa43c5acb2845a

Adjustments to the functionality of GoobiScript

After the GoobiScript runtime was completely re-implemented last month, some problems with GoobiScripts from plugins still arose that caused the plugins not to work properly. These problems have now been fixed and everything is working as usual again.

https://github.com/intranda/goobi-workflow/commit/d154874ba63a7e815267db378218c551cf11273c https://github.com/intranda/goobi-workflow/commit/a8320d942345e6aaa39bc856a1d701184c2ed0ae https://github.com/intranda/goobi-workflow/commit/d232a0f60d6db4b81c3c14d0ade4ddc1e668c97c

New functions and setting options of the dashboard

The enhanced dashboard has been extended even further. There are two new widgets: The widget Changes in the last 7 days shows steps that were completed, in progress or had errors in the last 7 days. The second new widget shows the steps that are currently in the queue. This is based on the new INFLIGHT status introduced last month.

Another new feature is that the layout of the dashboard can now be configured individually via the user settings. This configuration can be done by editing the setting Arrangement of dashboard elements within the user-specific User configuration in the tab General. Each line within the text box adds an element to the dashboard. Each line consists of two parts, separated by a space: First is a number that specifies the column in which the widget is to be displayed. The second part after the space is then the name of the widget to be added.

The list of currently available widgets is: assignedSteps, batches, htmlBox, itm, processSearch, processTemplates, queue, rss, statisticsProcesses2, statisticsProcesses, taskHistory, tasksLastChanges.

An example configuration could look as follows:

1 processSearch
1 assignedSteps
1 tasksLastChanges
1 taskHistory
1 htmlBox
2 statisticsProcesses
2 processTemplates
2 queue
3 rss

Coming soon

• Development of a new capture plugin for integration with Atlas AEON

• Development of a new plugin for editing configuration files from the user interface

• Development of a new plugin for editing rule sets from the user interface

• Development of a plugin for simultaneous editing of metadata from multiple processes

No new processes for deactivated projects

Until now, it was possible to continue to create processes within production templates for projects that are actually marked as deactivated. There has now been a change that should simplify the operation and make it more intuitive.

The buttons for creating individual processes and also those for mass import are now displayed in red for deactivated projects and show an explanatory warning message when used. In addition, the project name is also displayed in grey to indicate that the project has been deactivated. Deactivated projects are no longer listed in the drop-down fields that open by clicking on the arrow icon of the button.

Hiding buttons that are not required

Within the configuration of users, among other things, the required user groups and projects can be assigned that should belong to a user. At this point we have made a small optimisation that makes the operation more logical.

In both areas, the button for adding projects and user groups is now only displayed if further projects and user groups are available that have not yet been assigned. Accordingly, the button is now only displayed if it really enables a change. This should make operation a little more intuitive.

Improvements to the user interface for archive management

To improve the use of the archive management plugin once again, we have implemented some additional internal check routines and issue more informative messages within the user interface than before.

If, for example, the connection to the database is not available or expected information is missing, e.g. for the selection of the stock to be processed, helpful error messages are now displayed for the processor.

Adjustments to the menu

In the course of various internal menu changes, we have also merged the mobile menu (for smartphones and other narrow devices), which was previously maintained in a separate file, into one file. As a user, you will hopefully not notice anything. For us as developers, on the other hand, this approach is now much simpler, as new menu items to be added now only have to be considered in one central place.

We also took the opportunity to make some minimal visual adjustments and to change the colour highlighting of active pages. In the meantime, there was a colour inconsistency between some blue and some grey highlighted menu items. Now all active menu items are displayed in grey again.

New plugin to check ruleset conformity

We have developed a new plugin that we all didn't know we actually needed for a long time. The need came to light because large mass exports of data in particular occasionally resulted in errors due to rule set adjustments made in the meantime. Such situations can arise, for example, because metadata is defined differently at a later date. For example, if metadata is no longer allowed to occur but was allowed in the past, the METS files will not open correctly. Older Goobi processes were exported successfully and validly in the past, but with an updated rule set they may now violate the new rules.

This new plugin makes it easier to detect such processes. It can be easily executed for a filtered list of processes and checks the successful readability of the METS files at hand. In case of errors, these are listed in a table and it is possible to directly enter the METS editor for the corresponding process.

The source code of the plugin has been published here on GitHub:

The detailed documentation on installation, configuration and operation can be found at the following address:

New plugin for resetting pagination

In another project we had the special case that we had to completely reset the pagination for many processes, because the number of files for hundreds of processes changed. In this context, we noticed that there was no automatism in Goobi workflow that could make such a required change for many processes in mass.

For this reason, we have developed a plugin that allows resetting the pagination for large amounts of data. With the help of a freely selectable filter, the plugin can be used for thousands of processes.

The source code of the plugin has been published here on GitHub:

The detailed documentation on installation, configuration and operation can be found at the following address:

Display error in the user settings

If users had assigned many projects within the user settings, the display of the information was not optimal. We therefore made a small optical adjustment here in August so that even very extensive lists of projects are displayed correctly.

And because this representation is also useful for the user groups, we have also changed it. However, this small adjustment will only be included in the upcoming September release.

Change to the configuration

The storage of keystore information is now no longer done from the user interface of Goobi workflow. For a better use of keystores also by other programs the configuration has been changed and is now done via the central configuration file goobi_config.properties. A keystore can now be defined there as follows:

truststore = /path/to/file
truststore_password = password

These and other changes that need to be considered for updates have been included in the publicly available update instructions. Those who do not yet know them can find them here:

Database optimisations for larger data sets

Slowly but surely, some Goobi installations reach a data volume where database queries with unoptimised tables and database queries become too slow. For this reason, we have taken a close look at some database queries and tables and made changes. These adjustments are always included in the Goobi update. If, on the other hand, you carry out the update yourself, you will find further details on the changes in the update instructions:

Version number

The current version number of Goobi workflow with this release is: 21.08. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>21.08</version>
</dependency>

Coming soon

• New plugin for editing configuration files

• New plugin for editing rule sets

• Adjustments to the institution display

• AEON plugin

Print view for metadata revised

About two years ago we had implemented a completely new print view for metadata. This not only introduced the new feature of having a picture of the representative. Above all, it was now possible to print the hierarchy of the structure tree with the corresponding person and metadata.

However, what we had not considered in the implementation was that Goobi users will really use this function intensively right away, even for very deep hierarchies. We had initially only implemented the consideration of two hierarchy levels. With the current revision, however, we now go up to eight levels deep in the listing on the print view.

What is new in this context is that the corresponding pages are now also listed. For each structural element, both the image numbers and the logical page numbers are now listed as a page range.

More understandable display within GoobiScript

Once again there have been some minor changes within the execution of GoobiScript and the display of the results. A good example of such a minor change is, for example, that now the description of what the status of a GoobiScript is is a bit more meaningful. For example, for the GoobiScript for changing the project affiliation, understandable information is now displayed instead of the previously quite cryptic information that not even we as developers could do anything with. :)

Better documentation for the XMP header plugin

During the commissioning of the XMP header plugin on a productive system, we noticed that the documentation was not completely up to date. This has now been revised and in connection with this some changes have been made to the plugin, which allow even better logging in case the plugin encounters errors during execution.

Here you can find the current version of the documentation online:

Minor revisions to inline help

In some parts of Goobi workflow there were again several small changes to make it easier to use. These changes are too many and too small to be listed here. One example is the extension of the inline help texts, which have been slightly rewritten, including the configuration of file groups for the creation of METS files.

There have been several such changes in the background. In the best case, the user does not even notice these small adjustments.

Improved save behaviour in user administration and project administration

In the user administration, changes to the affiliations for projects and groups can now be undone with the Cancel button. This was not possible before. We have also made a similar adjustment to the projects, where the settings for the file groups now behave similarly.

Revision of the user interface for user groups

In the area of user groups, the interface for users was still not ideal. This mainly concerned the assignment of individual permissions that can be brought by individual plugins. The possibility of entering such individual permissions already existed. However, it was not intuitive enough, so that it was too easy to overlook the input option. This has now been significantly improved with a revision.

On this occasion, we have also taken into account that a user should be able to recognise even better when no further permissions can be assigned. In such a case, a short help text is now displayed.

Improved mapping of metadata when using Datacite DOI

A development on the DOI plugin allows us to now use metadata from multiple levels of a document and its parent unit in multi-volume works for registering DOIs, to pass them as XML elements to Datacite, or even to use them as a list of elements with attributes for registering DOIs.

Here again is the link to the documentation of the DOI plugin:

And here is the link to the source code of the plugin:

Clearer task log in the tasks

The task log is often helpful when it comes to understanding why a task is in which status, what previous communication has already taken place and what may have gone wrong.

Until now, the task log was only displayed as a small box within the tasks, which made it difficult to read the events comfortably when there was a lot of content. For this reason, the task log box is now expandable and takes up the entire width of the browser window. In addition, it displays all content in such a way that scrolling within the box is no longer necessary.

Version number

The current version number of Goobi workflow with this release is: 21.09. Within plugin developments, the following dependency must be entered accordingly for Maven projects within the pom.xml file:

<dependency>
  <groupId>de.intranda.goobi.workflow</groupId>
  <artifactId>goobi-core-jar</artifactId>
  <version>21.09</version>
</dependency>

Coming soon

• Outsourcing of some statistics to plugins with refreshing of the user interface

• Implementation of a new search syntax

• Revision of the internal folder structure to comply with the Maven convention

• Expansion of the plugin documentation

Conversion from Lido to METS/MODS and METS/MODS to Lido now possible

Goobi now supports the conversion of the metadata format LIDO to METS. For example, it is now possible to import a LIDO file and save it internally as a METS file. In the same way, an existing METS file can also be exported again as a LIDO data set.

https://gitea.intranda.com/intranda/ugh-extension/commit/e9ce2e551e644f9e20ab6caf69c08f60ec37b48d https://gitea.intranda.com/intranda/ugh-extension/commit/f3b6864a867c28567c8c76af89436daaa6fc65f4

Additional locations for image referencing in LIDO files.

Previously, images in LIDO were expected as <linkResource> within the <resourceRepresentation> element:

<lido:administrativeMetadata>
<lido:resourceWrap>
<lido:resourceSet>
    <lido:resourceRepresentation lido:type="local">
    <lido:linkResource>00000001.jpg</lido:linkResource>
    </lido:resourceRepresentation>
</lido:resourceSet>
</lido:resourceWrap>
</lido:administrativeMetadata>

With this month's developments, it has now become possible for images to be specified as <resourceID> instead. This is particularly helpful for data transfer from the MuseumPlus application:

<lido:administrativeMetadata>
<lido:resourceWrap>
<lido:resourceSet>
    <lido:resourceID>00000001.tif</lido:resourceID>
</lido:resourceSet>
</lido:resourceWrap>
</lido:administrativeMetadata>

Please note: If an image belongs to a sub-element, the specification must be made within the <lido> element that describes this sub-element.

https://gitea.intranda.com/intranda/ugh-extension/commit/183ef6f4b93a8b0becd0627df8094babce1a32ef

Adjustments to the usability of IIIF URLs in script calls

Recently, we had extended the so-called variable-replacer to the effect that it now also allows IIIF URLs for the corresponding images of a process in script calls. We had to make some corrections here so that the encoding of the URLs fits the respective use cases.

https://github.com/intranda/goobi-workflow/commit/b887afbc47b8871588e0c1f2d7275981e1141eab https://github.com/intranda/goobi-workflow/commit/1715a8603f476141b1c13d58385f51bc8641dff8

Building an Authority Server for Vocabularies