Basically, Goobi is a web-based application. To a very large extent, the way it is used and operated will be familiar to you from numerous websites. On the user side, all you need to be able to use Goobi is access to the internet and a web browser to take you to the Goobi site.

About this manual

Welcome to the Goobi Manual. This is the official documentation produced by intranda GmbH as the core developer of the open-source workflow management application known as Goobi. The entire body of documentation is divided into various sections. It is based on the experience we have gained over several years, both in the development of Goobi and during the installation and support process, involving numerous productive systems at establishments in various European countries. As well as providing answers to the questions most frequently asked by users, the manual explains typical scenarios in which Goobi can be used for different projects and examines the range of approaches adopted by different institutions.

Further development of Goobi and maintenance of this documentation

If you have any questions about this documentation, suggestions for the further development of this manual or general questions about Goobi, digitisation projects and, of course, the further development of Goobi, please feel free to contact intranda GmbH at any time:

Copyright

Please note that this document must not be amended or distributed in amended form. It must not be used for commercial purposes.

Please select on the left the chapter you are interested in.

Before we look in detail at the methodology of specific user groups, let us take a look at some basics. Any of the terms used in this section that may not be familiar to you can be found together with an explanation in the glossary.

To open Goobi for the first time, simply type the address of your local Goobi installation in your browser’s address bar, for example:

The page displayed by your browser should be similar to the one below:

All the official documentation relating to Goobi has been produced by intranda GmbH. It is divided into three main areas, each with a different focus on the same application, and aimed respectively at users, project managers and administrators.

Goobi User

The aim of this part is to provide a simple guide for Goobi users who do not have any specific responsibilities at administrator, project manager or other technical levels. Using examples to illustrate a wide range of typical scenarios, you will learn how to find your way around Goobi, perform basic tasks and complete individual steps in the workflow.

Goobi Managers

At manager level, the documentation takes a look behind the scenes of Goobi. You will find detailed descriptions of how to set up workflows and how to manage projects, users and user groups, as well as explanations of Goobi’s control functions, rulesets and other areas. This part is aimed primarily at those who are involved in the management of digitisation projects and who want to gain a deeper understanding of how Goobi works.

Goobi Administrators

The technical part for Goobi administrators goes way beyond the application’s actual functions. It takes an in-depth look at what goes on behind the user interface, with a particular focus on the configuration and fine-tuning of different settings, creating rulesets, specifying data imports, file system management, storage integration and other technical issues.

For security reasons, you should always log out of Goobi whenever you leave your work station. Depending on the configuration, Goobi will automatically log you out after a period of inactivity. To log out manually, simply click on your user name at the right-hand edge of the menu bar and select the Log out option from the drop-down sub-menu.

Once you have logged in, you can choose a number of settings to create your own user profile. To do this, click on your user name in the menu bar and then on the sub-menu item User configuration. Goobi will then display a form in which you can specify your personal configuration details.

In General view, you can choose a number of settings for working with Goobi. The Automatic logout after session timeout option allows you to choose how long you want Goobi to wait before logging you out automatically after a period of inactivity. In the diagram above, for example, the user has chosen to set this option at 120 minutes.

The value you enter in the Table size field (in this example: 10) determines how many rows Goobi will list on each page in a table before having to scroll on to display additional data on the next page of the table.

The Language for metadata option allows you to define which language from the configured ruleset you want Goobi to use for the structure data and metadata when you are working with the integrated METS Editor. You do not have to choose the same language as the one selected for the Goobi user interface, although it must be configured in the ruleset. This setting allows you to specify particular languages for structure data and metadata regardless of the languages in which Goobi itself is available.

The last of these user configuration settings is the Show process creation date checkbox. If this checkbox is selected, Goobi will display an additional column in the My tasks area and in the Processes and Production templates areas showing the date on which the process was created.

In the Mail Notifications section, you can configure which emails you want to receive when status changes occur in Goobi. The first step is to define the language in which the emails are written. The projects in which the user is a member are listed below the language selection. As soon as one of the projects has been clicked, it will be expanded and the list of tasks will be listed that are unlocked for the user groups in which the user is a member.

Here you can select the tasks about which the user wants to be informed in case of a selected status change. This is possible for individual tasks as well as for all tasks.

If the user has the user right Notifications for all status changes, all tasks are displayed instead of only those in which he himself is a member. In addition, notifications for further status changes can also be selected in this case, such as for error cases or the completion of processing.

For a detailed explanation of the other values you can enter in the user configuration section, simply click the Help icon on the menu bar. Goobi will display information about the options for each input field.

You will need to click Save in order for these settings to take effect.

In the top right corner you can switch between several available languages. Goobi is supplied and installed by intranda in German, English and Spanish. Adding other languages is straightforward, and this can be done at any time if required.

Goobi is the most popular open-source workflow management software for use in digitisation projects. It covers the entire spectrum of tasks involved in the digitisation process – from book to online presentation. Many libraries, museums, archives, publishers and even scanning companies have already adopted the software.

Goobi allows you to model, manage and supervise freely definable production processes and is used on a daily basis by many institutions to handle all the steps involved in creating a digital library: These include importing data from library catalogues, scanning and content-based indexing and the digital presentation and delivery of results in popular standardised formats.

Goobi meets enormously varied requirements while remaining as user-friendly as possible. It also needs to allow for simultaneous multiple use at a number of centres. As a multilingual web-based application, Goobi meets all these requirements. The following pages provide a detailed description of Goobi's functions and how they can help you coordinate your digitisation projects in a clear and organised way.

As is usually the case for web applications, you will find a navigation menu. After logging in, users can select from a number of options depending on their level of authorisation. The diagrams below show what the menu bar can look like after login depending on the authorisation level of the user.

As the illustrations above show, access to the functions offered by Goobi varies considerably depending on the level of authorisation of the user. Whereas some members of the project team may only be able to view their specific work, e.g. as a scan operator, users with a higher level of authorisation can choose from a much wider range of functions.

Users with a higher level of authorisation can view details of all the source works covered by those projects to which they have been assigned. They can also add new source works to Goobi based on Production templates.

Users with administrator rights have access to the widest range of functions and can view processes for all projects, not just for those projects to which they belong. They are also authorised to make changes to processes and workflows, add users, define user groups and assign individual users to user groups and projects. In addition to these settings, administrators can draw on a range of additional functions allowing them to configure projects in detail, specify rulesets with their structural and metadata and define the authentication process.

This section focuses on topics for those project members who work with Goobi on a day-to-day basis, for the most part through the menu option My tasks.

Towards the right-hand edge of the menu bar there is an option to call the integrated Help function. This can be used for many of the input boxes in Goobi and shows you which information is required and where. As well as an explanation of the content required in each field, the Help function will often provide examples.

Please note, however, that the Help function does not provide explanations for every area of Goobi. They are available only for input forms and boxes.

As with any other technical system, you should change your password from time to time for security reasons. To do so, click on your user name in the menu bar and then select the Change password option. This will display the following screen:

To change your password, enter your new password in the two boxes below. The change will take effect when you click Save. When you next log in to Goobi and from then onwards, you will have to use your new password. Do not disclose your password to any other users. In order to create a secure password, you need to use a mixture of special characters, numbers, both lower and upper case letters and at least 8 characters in total.

If you want to search for a particular process, you can use Goobi’s straightforward search box. To do so, select the Workflow - Search for volume option from the menu bar.

The input box used to search for a process in Goobi allows you to combine as many search parameters as you like. When you first open the input box, it will display the fields for the most common search criteria.

Each row in the search box contains several fields. The first of these allows you to specify which field you wish to search. The remaining fields relate to the value of the field being searched. In some fields (e.g. Process title and ID) you can enter a text-based search. In others, however, such as Project and Step, Goobi will offer you a choice of the actual search options available.

If the default number of rows shown for combining fields in a complex search is not sufficient, you can add (or remove) rows at any time.

To work with Goobi you will need a valid user name and password. You will usually receive these from your system administrator. In most case, they will be the same as those you were given for access to the operating system, for example. Please contact your system administrator or the Goobi support team at intranda GmbH if you have any problems with access or user authentication.

Once you have a valid user name and password, simply open the Goobi start page. Enter your user name and password and click Log in.

Once you have logged in, the Goobi screen will change. The menu bar will now contain a range of options for working with Goobi (depending on your authorisation level). Your screen will appear roughly the same as in the diagram below.

Your user name will appear at the right-hand edge of the menu bar when you are logged in. Click on your user name to display a menu with a Settings option. This will show you which user groups and projects you belong to. The other menu options available in the menu bar will vary depending on these user groups.

After logging in, most users continue with the menu option My tasks in the menu bar. This will display a screen similar to the one shown below:

In the My tasks list, you will only find those process tasks that form part of a project to which you have been assigned and which you are qualified to perform. In the diagram above, for example, a scan operator will only be shown tasks that are currently at the scanning stage of the overall workflow. The task name is shown in the Task column, while the name of the process to which the task belongs is shown in the Process title column.

The Project column displays the name of the corresponding project, and the Priority column provides a simple overview of the priority of the individual step. The priorities can be one of the following:

Explanation of the symbols for the priorities of a workflow step

Depending on your configuration and the process involved, you will also find additional information next to the status symbol that tells you how urgent the task is. If you see a priority symbol, this means that the project manager has prioritised this step. Tasks will always be shown in priority order regardless of which column you have chosen as the basis for sorting and listing your tasks.

As well as these priority indicators, there is an option to send error messages back from a later to an earlier point in the workflow. This means that errors detected at a later stage can be redirected backwards along with an error report. A red Correction button with a warning symbol in the My tasks list will immediately show the responsible person that the task listed in that row is one that was previously completed but that has now been found to contain an error. To display the error message, hold the cursor briefly over the red Correction button.

The status of each task can also be identified from the icon displayed in the Actions column. Within the My tasks area, you will come across the following Action symbols:

Explanation of symbols used for indicating the status of a workflow step

If you wish to check the meaning of any of these symbols, simply hold the cursor over the symbol in question. In the Actions column, for example, this might show you which user is already working on that particular task.

If the number of hits is very large, you can choose to navigate between different pages of the table. To do this, click on the arrow symbols below the table to move forwards or backwards. If you click on the current page number for the table, this will display an input box where you can enter the number of the page you want to view. Once you confirm this with the Enter key, Goobi will take you directly to the page you requested.

There is some scope for changing the way the list of tasks is displayed under the heading My tasks. As a general rule, you can arrange the tasks in ascending or descending order for any of the columns. If too many tasks are displayed, you can use the menu options above the table to modify the display.

If you wish to modify the display, the following options are available:

Options for customising the My tasks list

As well as the check boxes that you can tick to restrict the range of tasks displayed, Goobi gives you the option to filter specific tasks from the overall list of those process tasks that have been assigned to your project team and that you are qualified to perform. To display only certain tasks, enter your filter term into the Filter tasks input box. You can find a description of possible filter terms in the section Filtering processes. You can also save your filters and make them available for all other Goobi users.

This way, even complex filter terms can be selected and applied at any time from a list of previously saved terms. To save a filter and include it in the list of predefined filters, simply click on the save symbol after entering the term. You can then select and apply it any time you wish from the list.

If you want Goobi to display more columns in the My tasks list, you can choose from the following:

List of available columns that can be added to the task list

To adjust the column display, just select the column that you wish to add to the table display. To do so, tick the corresponding checkbox.

There is a specific screen for adding new processes to Goobi’s store of data. Select the option Workflow - Production templates from the menu bar. You will be shown a list containing all the production templates already available in Goobi together with their configured workflows.

This list gives you an overview of all those production templates that you as a project member are authorised to view. The only difference between this template list and the process list lies in the Actions column. To add a new process in Goobi, click the correct symbol in the Actions column.

This will open a new window allowing you to import the bibliographical data for the new process.

Once you have requested the catalogue, fill in all the fields marked at the end with an asterisk behind the label. These are compulsory fields, although you can of course fill in all the other fields, too. Do not enter anything into the Author-Title-Key (ATS) or the Title-Key (TSL) fields. The fields for Process title, Tif header document name, Tif header image description, Author-Title-Key or Title-Key are filled in automatically by Goobi when you save the new process or when you select the Generate button. The Author-Title-Key is a combination of some of the characters from the main title and the names of the people involved. The process name is made up of the Author-Title-Key generated by Goobi and a unique identifier. This way, each process within Goobi is given an unambiguous and meaningful name.

Before you click on the Save button, please ensure that at least one digital collection has been selected for this process. To select more than one digital collection, hold down the Control key. Finally, click on Save to move to the next page.

Goobi will now display an overview page that allows you to work with the new process you have just created. For example, you can generate a docket. To do so, just click on the Print docket link. To create another process, click Create new process. If you want to open the new process to modify its individual workflow or settings, click on Open newly created process.

Once a new process has been created, it can be viewed (together with its first open workflow step) by all those users and user groups who have been configured for that step. So, if the first step in the workflow is Scanning, for example, the newly created process will be visible to all scan operators assigned to the overall project containing that process. Accordingly, the task can be accepted and performed by any of these scan operators.

The methods used by scan operators working with Goobi on a daily basis do not vary substantially from those of other users with different qualifications or responsibilities. All users must first log in and select My tasks from the menu bar.

From an individual user’s perspective, working with Goobi on a daily basis generally involves selecting a task from the list of those offered and then clicking on the corresponding Actions button at the end of the row to view full details of that task.

The top-left box in the Details of the task window contains some general information about the task selected and accepted for processing by the user. You can use the Process log box immediately below this to enter whatever information you wish. This information will be visible to all users who subsequently accept other tasks as part of the same process at a later stage of the workflow. Its purpose is therefore to act as an open area for communication between different users in the form of general notes or observations. It can be used, for example, to draw attention to the fact that the volume in question needs to be worked on with particular care on account of its properties, or to a particular feature of the process that users at other work stations at a later stage of the workflow need to be aware of. Comments entered by users manually in the Process log are usually displayed in green.

It is also possible to upload files for this process, which are then listed within the process log. In contrast to the digital copies, these files do not serve to describe the work itself, but can contain information about the work or the method of operation to be observed. For example, it can be used to upload routing slips or offers from restoration service providers. The files uploaded in this way are available for all subsequent workflow steps, as are the comments.

In addition, you can switch to the file area at the top of the process log. Here the files of the different directories of a process are listed. Depending on the respective user authorization, these files can also be downloaded and deleted there. In the lower part of the process log it is possible to select a file to upload. After selecting a file, an optional description can be added. It is also possible to specify whether the file should only be used internally or whether it should also be considered for later export.

Information that has been transferred by external applications or scripts to the currently displayed process is displayed in different colors within the process log. The colors used here have the following meanings:

Colours in the Process log

To the right of the general properties box, you will find a number of extended task properties that can or in some cases must be entered by the user. These will depend on the configuration of your particular site, the process and the workflow step. In the screenshot shown above, for example, Goobi has been configured in such a way that the user must enter details of the opening angle and the scanning device.

Once a task has been accepted by the scan operator, Goobi will create a new folder within that operator’s work drive for storing the digitised files. Depending on the installation and configuration, users are generally allocated a network drive for use on the local work station computer. After a task has been accepted, that drive will contain an additional folder to which the scanned files can usually be saved directly.

Once a task has been completed, i.e. when all the required pages of the physical source have been scanned and saved in digital form in the folder provided by Goobi, the user needs to click on the Fishish this task link in the Possible actions area. This tells Goobi that the task has been completed. Goobi will then check for graphics files in the specified folder within the user’s work drive, check the names given to those files and end the task for the user. Goobi will also remove the folder provided for that specific process within the user’s work drive so the scan operator can no longer access it and the digitised material it contains. Whenever a user tries to close a task, Goobi will draw attention to any compulsory fields that need to be completed, e.g. the opening angle or the scanning device (see screenshot above). This means a task cannot be closed until all the required information has been entered in full by the user.

If, after accepting and starting a task, you decide that you do not want to work on the selected task, you can simply return it. Goobi will then reset that workflow step to its original open status. It is now available to any other authorised users or even to the same scan operator at a later stage. This arrangement can be useful if a task that has been selected involves too much work and cannot be completed, for example, on the last day before that user’s holiday, as it would otherwise remain unavailable to other users for a relatively long period.

Once you have closed a task, you will return automatically to the My tasks list. The completed task will no longer appear in the list. You can now continue with the next task from your list.

Once the reviewer has selected a task, Goobi will create a new folder within that user’s work drive (in the same way as it did for the scan operator) containing all the digitised material produced in an earlier step of the workflow (usually at the scanning stage).

In most cases, the workflow in Goobi is configured in such a way that quality control staff are not expected to make any changes within the new folder and have read-only access. Any general observations entered at an earlier stage are also visible to the reviewer in the Process log in the bottom left of the window. Any configured additional properties that users from this group need to enter or select during this task are shown next to the general properties. Using any standard image viewer, the user can now review each of the images in the new folder and check the quality of the digitised output.

If the quality matches project requirements, the reviewer will then click on the Fishish this task link to remove that task from the My tasks list. The new folder created by Goobi to allow the reviewer read-only access to the digitised material will then be removed automatically from that user’s work drive, thus preventing any further access to the data for that specific task within the specified workflow.

If a reviewer finds that the quality of individual files of digitised material is unsatisfactory, or that certain pages are missing or repeated, Goobi provides an option to send a correction message to a previous work station.

To do this, select the Report error tab in the Possible actions box. Goobi will display a list of all those tasks previously completed, together with details of the users responsible for those tasks. The reviewer can enter a description of the error found during quality control in the free text box below the selected task.

After the reviewer clicks on the link Send correction message, the incorrectly performed task will reappear in the My tasks list of the work station in question. The user will be able to view details of the error message by holding the cursor over the red warning message or by re-accepting the correction task. Having corrected the error, the user can then enter a description of the solution found. It is therefore possible to assign processes more than once within the workflow (in the event, for example, of an error).

On the page shown here, the details and plugins for tasks can be configured. The page does not differ significantly between tasks of process templates and those of existing processes. Differences are marked where appropriate.

Title

The title of the task is specified in this field. This can be freely selected. However, it should be taken into account that GoobiScript calls, for example, use the respective task titles to automatically execute background tasks across many processes.

Therefore titles should be short, meaningful and unique. Spaces and special characters are allowed.

Order

The position number is used to specify the position of a task in a process template or an existing process. Accordingly, an integer must be specified here.

On the one hand, the position number is used to display all tasks of a process in the correct processing sequence. On the other hand, when tasks are completed, those next tasks within a process that follow the sequence according to the current task are unlocked.

Several tasks can have the same position number. This then means for Goobi that the concerning tasks may be executed simultaneously. Parallel processing works with both automatic and manual tasks (for example, when several employees are working on a task in parallel).

If the task sequence of a process is configured with gaps (for example 1, 2, 3, 6), Goobi jumps directly to the task with the next highest number.

The order of tasks also plays an important role in GoobiScript calls. There, based on the number specified here, further administrative precautions can be taken.

Allow parallel tasks

This setting is only available when creating a new task within a process template.

Furthermore, this setting is only relevant when new tasks are to be inserted between already existing tasks.

If this option is set, the position number is set directly when the task is created. It may happen that another task already exists with the same number and both tasks can be processed in parallel later.

If this option is not set, then in the case of a duplication of the sequence number with that of another already existing task, the sequence numbers of the other and all subsequent tasks are shifted back by exactly one number when saving. This ensures that the new task inserted in the process does not duplicate any existing position number and that the subsequent tasks can still retain their defined numbering relative to each other (including missing and duplicated numbers).

If it turns out afterwards that this option was wrongly selected when creating the task and all subsequent tasks have a "wrong" position number, there are two possible solutions: For small adjustments, the position number can be adjusted at any time in the task overview of the process template using the buttons available for this purpose. For large adjustments, or if tasks already exist for the current process template, it is recommended to write a GoobiScript so that it adjusts all "wrong" numbers and is executed on all incorrectly numbered tasks.

Priority

In this list it is possible to select a priority for the current task. It should be noted that the priorities Standard, Priority, High priority and Highest priority are only intended to visually represent the importance of tasks. They have no further technical effect on the corresponding process.

The Correction option also has no effect on the task, but is automatically set if the final result in a task to be completed is not to be accepted and a correction message is sent.

Metadata

This option can be activated if it is in the sense of the task to edit metadata.

If a task is marked with the Metadata property, additional icons and options are displayed at various places in the user interface to access metadata. For example, the button for calling the metadata editor is also displayed if a task has this property.

If metadata in a task is to be uploaded, downloaded, validated or used in any other way, this option must be selected.

Read images

This option can be enabled if the user in this task should get read access to image files in his user folder (on the Goobi server). This may be the case, for example, when images are to be downloaded or displayed for quality checking.

Write images

This option can be activated if the user in this task should get write access to image files in his user folder (on the Goobi server). This is required whenever images are to be uploaded or edited.

Validate on exit

This option can be enabled to validate the metadata of the process when completing this task. This validation has nothing to do with the validation plugin (see below). The type of validation set here checks if all metadata, structure elements (DocStructs) and page counts have been applied according to the rule set.

Export

This option can be enabled if it is in the sense of the task to export data for further processing with other systems. This can be, for example, other database formats, content management systems (CMS) or simply certain file formats. If this task is defined as an export task, an export plugin must be selected in the 'Step plugin' field. Export plugins usually start with the prefix intranda_export_.

If this task is an automatic task and an export task at the same time, the export will take place automatically. Regardless of this, the user will see an export button next to the corresponding task in the overview and can also export the dataset manually.

Skip this task

This option can be selected if the task should be skipped in the process. If a task has this property, it will be closed automatically as soon as a user accepts it. If this task is an automatic task, it will also be skipped and closed automatically.

Automatic task

Different types of tasks can be configured as automatic tasks. This allows, for example, plugins or GoobiSript calls to be executed either directly or in one of the available processing queues.

Specifically, the following types of tasks can be automated: Internal tasks (intranda_step_*), export tasks (intranda_export_*), script tasks, HTTP tasks and time delay tasks (intranda_delay_*). In each case, make sure that the appropriate plugin or script is selected as described in the corresponding chapter.

In order to use processing queues, they must first be enabled and set up in the corresponding configuration files (goobi_config.properties and goobi_activemq.xml). If this option is then activated, a drop-down menu appears below the checkbox from which the desired processing queue can be selected.

If the option Don't execute in processing queue is selected, the corresponding plugin or script will be executed directly as soon as the task is reached in the process. This option is recommended only for tasks that should be triggered by the user in real time, for example by completing the previous task.

The Processing queue for fast jobs and Processing queue for slow jobs options provide two independently operating processing queues that are normally the right choice for most automated tasks. The fast processing queue is intended for tasks that have a rather short execution time and should be completed promptly.

The slow processing queue, on the other hand, should be used for tasks that require a lot of processing time and for which it is not really relevant how quickly they are completed. For example, the slow processing queue can be used for large amounts of image exports, OCR analysis, 3D calculations, or other complex applications. As a result, this processing queue is also suitable for tasks that sometimes require a total computing time of hours, days or even longer over many thousands of processes.

However, by default, Goobi always prioritizes tasks that are being executed by users in real time and tasks that are in the fast processing queue. For example, if Goobi is busy during the day due to the work of many active employees and there is also a well-filled slow processing queue, it is therefore common for this to make its greatest progress at night.

In addition, there is an In queue for external processing option. This processing queue can be used by REST API requests. Suitable REST API requests are usually provided by plugins.

Generate thumbnails

If the task is to be used to generate thumbnails, this checkbox must be selected. A text input field then appears, in which an example configuration for the generation of thumbnails is given. This should be adapted for the project's own needs.

The text input field includes several lines in which a YAML-compatible notation of key-value pairs is expected. Key-value pairs are separated by a colon (:) and each line may contain exactly one key-value pair. The first line contains the string --- to indicate the beginning of the data set. Comments are marked with a hash (#) and may be placed in a separate line as well as at the end of a line used for content. They are ignored by the interpreter. Thus it is also possible to "comment out" certain parts in experimental configurations.

At the beginning the following example configuration is in the text input field:

The variables have the following meaning (details, see below):

Note that Sizes accepts a list. Each line starts with the string - and then contains an integer. The list entries must directly follow the line Sizes:. Text values must be enclosed in double quotes ". Boolean values can be used directly and can be set to either true or false to turn the feature on or off.

Master

Setting this value to true will generate thumbnails for all image files inside the master folder (usually /opt/digiverso/goobi/metadata/{processId}/images/{processId}_master/) and store them in the thumbnails folder (usually /opt/digiverso/goobi/metadata/{processId}/images/thumbs/{processId}_master_{size}/).

Media

Setting this value to true will generate thumbnails for all image files inside the media folder (usually /opt/digiverso/goobi/metadata/{processId}/images/{processId}_media/) and store them in the thumbnails folder (usually /opt/digiverso/goobi/metadata/{processId}/images/thumbs/{processId}_media_{size}/).

Img_Directory

At this point, a third, additional folder can be specified for the generation of the thumbnails. The generation behaves here in exactly the same way as with the master folder and the media folder. However, this folder must be specified as a full folder path, for example /opt/digiverso/goobi/metadata/{processId}/images/{processId}_custom/. The destination folder for generation would then be accordingly: /opt/digiverso/goobi/metadata/{processId}/images/thumbs/{processId}_custom_{size}/. If the folder path specified here is empty or the parameter is missing, this option will be ignored during generation.

Custom_script_command

A path to an individual script file or to an executable program that is to be used to generate the thumbnail files can be specified here. The script or program should be able to independently recognize the image files and use the correct process-related folder paths. If the field is left empty, Goobi will use internal Java libraries for generation.

Note: This option is not currently supported. All thumbnails are generated using special image processing Java libraries, regardless of any script or program specified here.

Sizes

At this point a list of image file sizes (in pixels) can be specified in which the thumbnails should be generated. This can be done by specifying one or more lines, each of which starts with the YAML list entry prefix (- ) and then contains an integer. Note that each of the specified image file sizes will be used independently, and will always produce thumbnails that have the specified size on their longest side. For example, if the values 400 and 800 are specified, both thumbnails with a size of 400 pixels and thumbnails with a size of 800 pixels will be generated.

Additional technical information

• In the examples above, the placeholders {processId} stand for the respective process ID and {size} for the thumbnail size in pixels. These are inserted in folder names accordingly.

• All settings (except Sizes) can be omitted, in which case the default values given above will be used.

• If multiple thumbnail sizes are specified, all will be generated independently. All respective sizes are stored in their individually named folders (consisting of image size and source folder).

• All generated thumbnail images are JPG files using a default color profile of the image file processing Java library.

• Watermarks are not included in preview images.

• Spaces in filenames of original files are replaced by %20 in thumbnail image file names for technical reasons.

• If the folders for the preview images do not exist, they will be created automatically.

• If the output folders including preview images already exist and if the source image files have not changed (measured by the last-modified value of all involved image files), the thumbnail images will not be regenerated.

Script step

This option can be enabled to run one or more scripts on the server as part of the current process. For example, these can be Bash or Python scripts that perform background tasks or interact directly with the current task's dataset in the file system.

The numeric return values of the scripts play an important role in workflow control. Therefore, they are documented in detail following this chapter and should be strictly observed.

If the script step option is activated, a table appears below the checkbox in which up to five script files can be entered. For each script a name and a path can be entered, whereby the path is the absolute path (including script file name) in the server file system. The associated name can be freely selected and serves exclusively the simpler recognition for the user.

Any fields in the table can be filled in or left blank. If this task is executed later in a process, Goobi will execute all filled script path fields from top to bottom.

Care must be taken to ensure that the specified scripts are executable. Otherwise, error messages in the journal and server log files will indicate the cause of non-execution. Common causes of errors are for example missing execution rights, wrong environment variables (especially PATH), missing interpreters, wrong file paths or missing parameters. Parameters can easily be specified in the script field.

If another script language is to be used, an interpreter must be specified if necessary. In this case, the interpreter is the executed program and then executes the script file specified as a parameter.

The script lines read from the table (but not the script files themselves) will be searched again by the variable replacer in Goobi Workflow when executing the scripts in a process and any variables will be replaced. This can be used to insert metadata values into certain fields. For example, parameters for the scripts specified here can be written so that the parameter values correspond to the variables for the variable replacer.

A few examples of scripts:

If the available script fields are not sufficient, a kind of meta script file can be written instead, which in turn executes several scripts and occupies only one field in the table.

Return values of script files

The respective return value of a script is used by Goobi Workflow to determine the current status of the corresponding task, especially between the processing of multiple scripts within one task. A distinction is made between Success, Error, Reopen task and Continue task.

The return value, for example 99, is specified in Bash scripts with the command return 99 or in python with sys.exit(99) to the end of the script. This should properly handle any errors caught by if blocks or other situations, and in particular should not use return statements without return values.

The following table gives an overview of the respective behavior of Goobi:

If a script outputs text in the error output stream (STDERR), this is also treated as an error condition (1). Errors are documented in the journal and server log files.

The input and output streams STDIN and STDOUT are currently not used.

Combination of multiple scripts

The value 0 can be used if a script is the only or last script in a task. If the script was successfully executed, the task is set to Completed status. If 0 is used and scripts follow afterwards, unwanted side effects may occur.

The value 1 (and all undefined values) can be used if a script fails. Goobi will then not execute any further scripts and set the current task directly to Error status. Error messages (STDERR output and goobi-internal errors) are logged in the journal and server log.

The value 98 can be used if a script could not be executed as desired and Goobi should be told to restart the whole task. Accordingly, the execution of the first defined script will be started again if it is an automatic task. The status of the task is set back to Open by the return value 98 and to In process in case of an automatic task. This return value can be used, for example, for situations when scripts perform error handling in their own error detection and then need to be re-executed to benefit from their own error corrections on the next execution attempt. Validation scripts can also be used at this point to detect errors from previous scripts and cause the task to restart accordingly.

The value 99 can be used if a script could be executed successfully and another script is to be executed afterwards. The status of the task is not yet set to Completed with 99 (in contrast to 0), which makes sense if several scripts follow each other. The last script should then return 0 to complete the task.

If the provided script fields are not sufficient and multiple script calls are outsourced to an external script file, make sure that all return values are passed correctly and arrive back in Goobi to achieve the desired effects in task automation.

HTTP step

This option can be selected if the current task is to perform an HTTP (or HTTPS) transaction with an API of another server. This application is mainly intended for POST, PUT and PATCH requests, i.e. for uploading data to the respective services. However, GET requests can also be sent conditionally to check the availability of resources. As soon as the corresponding checkbox is selected, several input fields appear in which the details of the request can be entered.

With the option `Close step after successful HTTP call, it can be configured whether the task should be closed automatically in case of a successful HTTP request. More details, see below.

First, the HTTP method must be selected. The method to use is determined by the API endpoint used (usually within a REST API) and must match the method specified in the associated specification. Otherwise, the accessed server can be expected to return an error message (405 Method Not Allowed, see below).

In general, APIs should adhere to the following request method conventions, although deviations are possible:

* If unchanged properties are omitted, this may lead to unintended deletion of the data in question, since the API endpoint assumes that the properties should be overwritten with "empty" data. Details on the specific behavior of the API should always be found in the associated specification.

** Note that the GET method is only partially supported at this point. The received data is not stored or further processed. The GET method can be used to query the existence or accessibility of data.

The next step is to specify the HTTP URL. This is composed of the domain name or IP address of the server, optionally the port number, the API endpoint and the required URL parameters. If the default HTTP port '80' is used, this can be omitted.

Some examples of API request URLs:

After that the HTTP body is entered. This is only relevant if it is a POST, PUT or PATCH method. The content of the request depends on the API used and the corresponding API endpoint, so we cannot go into more detail here. However, if a REST API is used, data must always be entered in JSON format.

The specified data in the HTTP body, once the HTTP request is later performed for a process, will be post-processed by goobi's internal variable replacer. This makes it possible to use metadata field labels according to the variable translator's syntax rules in the request content. These are then replaced by the actual data in the respective process before the request is sent.

Additionally, the Escape HTTP body as JSON option can be selected. This modifies control characters (for example, line breaks or tabs) in the request content so that they are masked by backslashes (\\) and can be used correctly by the API endpoint if necessary. However, the exact handling of such control characters also depends on the API endpoint and must be read in the associated specification or requested from the provider.

This completes the settings for the request.

As soon as the request is sent later during the execution of the task within a process, the requested server responds with an HTTP status code. This is read by Goobi and decides whether to continue executing the process. In general, successfully answered requests lead to the completion of the respective task. If the next task in the process is an automatic task, it will be executed.

With the option Close step after successful HTTP call, it can be configured that the task is really closed if the HTTP call was successful. If this option is not choosed, the task will remain in the status Open.

If an error status code is returned, the task remains in an error status and manual intervention is required. Errors are logged in the journal and server log files.

In the realm of HTTP status codes, normally all status codes < 400 are considered successes and all >= 400 are considered errors.

The following is a listing of the most common status codes associated with REST APIs:

Successful status codes:

• 200 OK - The request was processed successfully or the data set could be found

• 201 Created - The resource was created successfully

• 204 No Content - The request was processed successfully, the answer intentionally does not include data

Error status codes

• 400 Bad Request - The request was not formatted correctly - The syntax should be checked

• 401 Unauthorized - Authentication is required for this request (or the specified authentication is invalid)

• 403 Forbidden - This resource exists, but may not be requested

• 404 Not Found - The resource does not exist - possibly the URL or URI is incorrect

• 405 Method Not Allowed - The wrong HTTP method was used (see above)

• 429 Too Many Requests - Too many requests were sent within a certain period of time

• 500 Internal Server Error - The server has detected an internal error for which the request is not directly responsible

• 501 Not Implemented - The API endpoint does not exist - Possibly an incorrect API version is used

In addition, some further status codes can be returned. The respective meaning can be found in the relevant RFC standards. In case of special problems, however, the responsible support should be contacted.

Status

In this drop-down menu, the status of the task can be set manually. Typically, in a process template, all tasks are set to Locked and the first one to Open, so that later in each process, editing can start at the first task. In the course of editing, Goobi sets all open tasks to In queue or In process and finished tasks to Completed. In case of an error, the task is automatically set to Error.

Additionally, there is an option Deactivated to deactivate a task completely. This will be skipped by Goobi when editing the process. The Error status can also be used manually to indicate that an edit has not taken place as desired and may need to be repeated.

When setting the status in a task in an already existing and edited process, care should be taken here to avoid unwanted side effects caused by automatic tasks being opened or closed manually, which are triggered by this intervention in the workflow.

Batch step

This option can be enabled if the current task should be able to be grouped and executed together with similar tasks within a batch.

Step plugin

In this drop-down menu, a plugin is selected that is to be used for the task currently being processed. There are different possibilities for what the plugin selected here can be used for. Different possibilities are described in more detail in other subchapters on this page.

Plugins can be used for automatic or semi-automatic tasks. In this case, time delay plugins (intranda_delay_*), validation plugins (intranda_validation_*), export plugins (intranda_export_*) or other types of plugins can be selected. Plugins that bring their own user interface and are not already described in one of the other categories on this page can also be selected here (e.g. intranda_step_*).

For tasks that should not use a plugin, this field is left empty.

Validation plugin

A task can be configured to run a validation plugin upon completion. The corresponding plugin can be set in this field. Validation plugins usually start with the prefix intranda_validation_.

Plugin for time delay

Time delay plugins can be included as tasks in the process to pause automatic processing for a certain time. If a task is to be used for time delay, this checkbox must be set and a corresponding plugin must be set.

The corresponding plugin is selected in the Step plugin field. Plugins that can be used for time delay usually start with the prefix intranda_delay_.

Update metadata index when finishing

This option can be enabled if metadata has been modified by completing this task and the search index should be updated for the entire dataset of this process.

Download the docket as PDF file

This option can be enabled to offer generating and downloading a docket PDF file for the current intermediate status of the corresponding process in the task overview. This can be handy to manually check the current technical status of the process.

In addition to the tasks we have already described, it is possible to configure Goobi so that users can call up any external programs, scripts or plugins for specific purposes from within Goobi. To this end, Goobi provides an option to configure additional buttons in the Possible actions area for individual steps of the workflow. The names of these buttons are specified by the Goobi administrator when configuring the workflow. When they are activated, they call up one or more configured instructions on the server.

Based on the way Goobi has been configured by the Administrator, the user can decide whether the scripts are actually executed or not. The user simply needs to click on one or more of the proposed actions. Depending on the configuration, the user will receive a message to confirm that the server has executed the script correctly.

As well as the manual scripts described in the last section, Goobi can also be configured to execute individual workflow steps automatically if they have been configured to call up scripts.

To do this, as for the manual scripts, the administrator will assign a series of different shell commands and stipulate that they should be performed automatically one after the other in the specified sequence. Automatic script-run steps are not generally included in a user’s task list. If an error occurs during the execution of the configured server-based script calls with the result that the script cannot actually be executed, the automatic workflow step will keep the same status and will be displayed automatically in the assigned user’s task list. Automatic script-run steps are therefore only visible in the user’s task list if errors occur during execution.

To identify the error, the user can employ the same method as that described above for manual script-run steps, i.e. by starting the scripts in question manually and thus identifying the source of the error. In cases where the automatic scripts make use of the function allowing users to send information or error messages concerning a specific process, the error messages are also listed in the Process log within that task without having to run the script again to identify the error.

Automatic script-run steps that successfully execute the configured scripts automatically close the current workflow step and activate the next step in the workflow.

The next section examines some typical user groups in order to illustrate their structure as part of digitisation projects at numerous establishments and the way they work with Goobi on a daily basis. In terms of the way tasks are performed, any user groups or methods that may be in place at your own establishment but not specifically represented below will nevertheless work along similar lines to those shown in the examples. Accordingly, all the examples given are also applicable to individual workflows and other project-specific arrangements.

All the processes that you or other users create will remain in Goobi’s database. To display an overview of these processes, select the option Workflow - Processes in the menu bar.

Goobi will display a table of all the processes that you are allowed to access. This will depend on whether you are a member of those projects or an authorised Goobi administrator. These details are stored in your user account and will determine how much of the stored data will be displayed in the Current processes table. Equally (unless you are an authorised administrator), Goobi will not display processes belonging to projects that have been marked as completed by an administrator.

The process list gives you an overview of all those processes that have not yet completed all the corresponding steps. In addition to these active processes, if you wish to display processes that have completed all their steps, select the table menu option Modify display and check the Show finished processes box. This will display all the Goobi processes that you are authorised to access regardless of their current status. If you are a Goobi administrator, by ticking the checkbox Show deactivated projects you can also list those processes for which the overall project has already been marked as completed.

The overview of current processes allows you to work with specific processes and modify their data or the status of individual steps independently of the workflow. To do this, you do not yourself have to be a member of the user group that is registered in Goobi as the group with responsibility for the individual workflow steps making up the processes in the list. This means that you can, for example, access Goobi’s integrated METS Editor regardless of whether the status of the current process indicates that it is at the stage of obtaining and recording structure data and metadata. From this screen, you can also export the data to other systems, e.g. a presentation system for digitised material such as the intranda viewer. You can also, for example, generate PDF files from the digitised material in combination with the structure data and metadata that have been recorded.

Each of the columns in the process view table can be sorted into ascending or descending order by clicking on the column headings. The little sort direction symbols at the right-hand edge of the column heading cell will change accordingly.

The Process title column contains the unique identifier for each process. It is generated automatically from the metadata whenever new processes are added to Goobi.

The Status column shows you immediately how far a particular process has advanced in the workflow. The colour system used here indicates how many steps have already been completed out of the workflow by those users involved. The red section of the bar shows how many steps are still locked and cannot yet be processed by users. The yellow section shows how many steps are currently being processed or waiting to be processed. For details of the status of individual process steps, just click the process in the Process title column. Goobi will display a detailed list of all the steps for that process and their current status.

One of the workflow steps frequently included in a wide range of digitisation projects at numerous establishments involves making the digitised material available to the public together with the corresponding structure data and metadata, obtaining which can sometimes require a considerable investment of time and effort. This digitised material can be made available in a range of systems independently of Goobi and can be used, for example, on completely different hardware from that required to run Goobi. In this step, Goobi will export the digitised material, together with the structure data and metadata in the form of a METS file, to a Document Management System (DMS). This step can either be completely or partially automated, depending on the configuration.

If the procedure is only partially automated, the data is exported manually by the responsible person in the user group using roughly the same methods as those described earlier for the other user groups. After logging in and selecting a task from the My task list, the user will be shown a screen containing the details of that task. As in the previous examples, in addition to the task details, this screen contains a Process log for messages about the corresponding process. However, an additional button can be found in the Possible actions area. This button can be selected to launch the actual export to the external DMS system.

If Goobi has been configured by the system administrator in such a way that the user does not have to wait until communication with the DMS has ceased, it will display a positive message immediately (after the user clicks the Export to DMS link) to indicate that the export process is now running in the background. The results of the export are still validated in the background (even though validation is not displayed on the user screen) and are stored with details of system events in the form of log files. This means that the user does not have to wait and can continue to work with Goobi immediately by clicking the Finish this task link. The task is then removed automatically from that user’s My tasks list.

From Version 1.9 of Goobi onwards, users have been able to perform a fully automated DMS export in addition to the manual option. In this case, DMS export tasks will only appear in a user’s task list if an error has occurred during the export process. Assuming that Goobi has been correctly configured, the validation of earlier workflow results should ensure that the automatic export is successful. In such cases, the export process is fully automatic within the workflow and will on completion activate the next task in the workflow.

In Goobi 2.0, the Export function has been updated so that users can integrate various plugins at this point allowing them to respond flexibly to the requirements of different presentation environments. If you wish to use this plugin-based export function, the workflow step must be configured with the Export DMS setting. However, you will also need to enter the plugin that has been provided. If the special plugin-based export function has not been configured for that workflow step, Goobi will use the default export function.

The grey-shaded area on the left of the screen contains the structure tree, where you can see in hierarchical form all the structure elements that have already been obtained from the source material. When you select a structure element, it will appear in bold in the tree view. For each structure element, the descriptor is based on the type chosen for that element.

Click on the icon just in front of the structure element to expand or collapse individual sections of the structure tree. If you hold the cursor over the small symbol next to each structure element, you will see a pop-up with further details of that structure element without having to open it.

Goobi’s Metadata Editor is one of the core elements of digitisation projects. Most work involving the Metadata Editor is performed by trained librarians. There are two ways to open the Metadata Editor. Administrators and project managers can open the screen for processing structure data and metadata at any time from any individual process listed in the process view page, regardless of its workflow status.

Users with a more restricted level of authorisation can access the Metadata Editor through the My tasks area but can only perform those workflow steps for which they have been designated as the responsible user.

Once you have opened the Metadata Editor, you have full access to all the editing options relating to the pagination, structure data and metadata of the digitised material. The Metadata Editor is divided into a number of sections.

The right-hand side of the Metadata Editor gives you an overview of the individual digital images that form part of the current process together with the number of pages, the current magnification level, the currently displayed image number and information about which derivative of the available digitised material you are currently viewing.

Page navigation

You can move between individual pages using the Forward and Previous links just above the image. You can also move quickly to the previous and next images by selecting them directly.

The current page is shown in the middle of the page range with the previous and next two pages on each side (provided they are available for display). Clicking on any of the pages will take you directly to the corresponding scanned image. Goobi also features a number of keyboard combinations for repeated navigation between different pages of the source material.

Keyboard combination for navigating between images in the Metadata Editor

Using these keyboard combinations allows you to move quickly and easily through the digitised material, even over large areas. Another navigation option allows you to move directly to a specific image by entering that image number in the Go to image box and pressing return. Goobi will then automatically display the requested image.

Zoom

In addition to all the above navigation options, you can also change the way Goobi displays the scanned image. To do so, just click on either magnifying glass symbol below the page navigation bar to increase or reduce the magnification/zoom level. The current magnification is shown between the two magnifying glass symbols. If you want to select a particular level of zoom, simply click between the two magnifying glass symbols, enter a figure and press the Enter key to confirm.

Rotation

Goobi also allows you to rotate the image in 90 degree stages to view information that can only be read in landscape format. To do so, just click on the arrow pointing to the right to rotate the image clockwise and on the arrow pointing left to rotate the image anti-clockwise.

Selecting the image folder

Underneath the image you will find an option to select from different derivatives of the image (where available). In the drop-down list entitled Folder, Goobi lists all the image folders linked to the currently selected process. For example, if you already have a number of derivative images for a specific process (e.g. master images, scaled or compressed versions of images or derivatives with a different tone such as bitonal images), you can switch between these derivative images simply by selecting the corresponding folder.

The main area of the Metadata Editor also contains a navigation bar to perform a range of steps during metadata indexing. Selecting Pagination, Structure data, Metadata and File replacement allows you to switch between the individual editing modes in the Metadata Editor. Each of these modes is explained in greater detail below and should be performed in the specified order.

Goobi offers a wide range of functions that simplify the task of working with structure data and metadata. These options can be found in the top menu bar and at the right-hand edge of the structure tree pane in the Metadata Editor.

Functions in the top menu bar

The following functions are available from the top menu bar. The corresponding icons can be found on the right-hand side of the bar.

Validation

At any time while you are working with the Metadata Editor, you can use the Validate option to check whether you have complied with the rules provided by Goobi for the structure data and metadata. These rules are defined individually for each project by the Goobi administrator. Before closing the step you are working on, Goobi will conduct a further validation check. If this validation fails, Goobi will not let you close the task. It is therefore advisable to perform regular validation checks in the Metadata Editor to ensure that you have complied with the rules. If you click Validate and Goobi then determines that your metadata and structure data do not comply with the established rules, it will display an explanatory warning message to that effect.

Fixed image

By choosing this function you can prevent the image from scrolling in the Metadata Editor when you are dealing with long lists of metadata or during the allocation of pagination sequences. If you activate this function, the selected image will remain in place on the right hand side of the window together with its navigation bar and the zoom options, even if you scroll up or down on the right of the Metadata Editor.

Settings

This menu bar icon provides several options for additional settings. Simply click the icon to display the available functions.

Save

The Save option allows you to save your work in the Metadata Editor at any time. You should always click on this button when you want to make sure your data cannot be lost.

Exit the METS Editor

Click this menu bar icon to exit the METS Editor. You then have two options.

The Exit option exits the Metadata Editor without saving your work, so you should usually ensure that you save your data before selecting this option. This function can also be used, for example, to exit the Metadata Editor without having to save your work in the event that you unintentionally delete or change the data.

The option Save and exit saves all your work and closes the Metadata Editor. Goobi will then return you to the screen from which you launched the Metadata Editor. If you accessed the Metadata Editor from My tasks, for example, you will find yourself back in that area. If you launched the Metadata Editor from the Processes area, selecting Save and exit will automatically return you to the same point in that area.

Additional settings

If you select the menu bar icon Settings, Goobi will display a drop-down list containing several additional functions.

Display the hierarchical level in the structure hierarchy

Click on this function to display information about the hierarchical level of each structure element in the structure tree on the left. This function is particularly useful in cases where the hierarchy of the structure tree contains many levels. An indication of the hierarchical level can help you determine which level you are currently viewing.

Fully collapse the structural hierarchy

Using this function, you can switch between the fully expanded and fully compressed versions of the structure tree on the left hand side of the Metadata Editor. This allows you to fully expand even the most complex structure trees with a single click without having to click and expand each section within the tree individually.

Hide image / Show image

You can select the Hide image option at any time to prevent the image from being displayed. Conversely, the Show image option causes the image to be shown again after being hidden.

Reset pagination completely

This option allows you to recommence pagination. You may wish to use this function to reset all the pagination data and start indexing the pagination sequences again from the beginning.

Show physical structure tree / Show logical structure tree

This function allows you to work with the physical structure tree rather than the logical structure tree. This allows you, for example, to record metadata for individual pages or edit metadata stored in the physical metadata area.

Functions available at right-hand edge of the structure tree pane

The structure tree pane contains a number of additional icons/functions that can be used to modify the way the structure tree is displayed.

Link the image with the selected structure element

The function gives you an option to link the structure tree on the left of the Metadata Editor to the image. If this function has been activated, the image display will change to the start page of the structure element you have selected in the structure tree.

Show main title in the structural hierarchy

If you activate this function, Goobi will display the names given to individual structure elements directly within the structure tree. This may give you a better overview, e.g. to help you locate chapters by their name in the structure tree.

Display the assigned pages in the structure hierarchy

Select the Tree page number to display the page ranges allocated to each structure element in the structure tree on the left. The first and last page of the range will appear in brackets next to each structure element. Each page range displayed contains the image number and the page label (printed page number).

Print this page

Select this icon if you wish to print out the structure tree.

When you have completed the task of pagination, the next step in the workflow involves structuring.

Working again in the Metadata Editor, the aim is to identify and record all the relevant structure elements of the material in a tree-like list. In this step of the workflow, all the structure elements to be indexed for that source material are marked using as precise a descriptor as possible and allocated to the corresponding structure element within the hierarchy. By way of example, for a monograph this can mean that several chapters, a foreword, an afterword, an index and other structure elements can be added under the structure element Monograph. Further sub-elements can then be added under these structure elements and so on. In fact, the Metadata Editor allows you to create a hierarchy of elements with as many levels as you wish, including sub-chapters within chapters.

To start the task of organising the structure elements using the Metadata Editor, first select the Structure data option from the navigation bar. Goobi will display an overview of the available options. It is important to remember that all changes to the structure data depend on which structure element is currently selected in the structure tree on the left (highlighted in bold).

The list of elements from which you can select in the New structure element box will vary depending on which structure element has been activated. You can choose the position where you want to insert the new structure element from the following options:

Positions for new structure elements

The list of structure elements from which you can select will vary depending which of the four position options you choose. The range of elements available for selection depends on how the Goobi administrator has specifically defined the system for the current project using the freely configurable rulesets. Within certain structure elements, this means that structure data and metadata can be specified that are admissible within a given hierarchy. For example, within the structure element Monograph it is not possible to assign the structure element Periodical article, although you could assign a Chapter, which is a perfectly normal component of a monograph. Depending on this configuration, the list display will be based on the structure element currently selected in the left-hand navigation tree.

Structural elements for pages

The fastest way of adding new structure elements in Goobi is to choose the option As last sub-element As last sub-element First page input box. Move forward until you come to the last page for the structure element you wish to add, and this time click on the image symbol next to the Last page input box. Clicking the image symbols instructs Goobi to apply the page number currently selected on the right of the image display. Using the little arrows on either side of the image icon, you can enter the page number for the preceding or following page. This allows you to allocate pages without spending an inordinate length of time moving between pages in the image display section. Next, click on Add structure element to insert this new structure element as the last sub-element within the current structure element. This is an efficient method of adding structure elements.

If you want to create additional sub-elements as part of a structure element that you have already added, first select the existing element to which you want them to belong. You can now use the same method as described above by adding each new structure element as the last sub-element.

This single step of the workflow therefore involves providing two important pieces of information. As well as identifying and recording the digital structure of the source material (to reflect its logical structure in the form of chapters, sub-chapters, forewords, afterwords, indices and any other structure elements that can be configured individually in your Goobi installation), when you add structure elements you are also providing the corresponding page ranges for each one of those elements.

Clicking on the small arrow symbols next to the First page and Last page text boxes instructs Goobi to set these as the beginning and end pages for that structure element. As the metadata file grows in the background, it will eventually be perfectly clear which page ranges correspond to which structure elements. If the digitised material is later made available for viewing, e.g. on a website, researchers will then be able to display all the pages corresponding to a particular sub-element, and sub-chapters downloaded in pdf format will always contain the correctly allocated pages.

For some structure elements, Goobi will display input fields below the selected type so that metadata can be allocated directly to the structure element. Which fields are displayed (e.g. main title, keywords) will depend on the structure type that has been selected and on the ruleset configurations in Goobi.

Structure elements for image areas

If structural elements are to be assigned to image areas, this can be done in a similar way. However, instead of clicking on the image icons for the start image and the end image to be used, select the icon below for the image section.

Now use the mouse to select the area on the displayed image that you want to define for the structural element.

You can then proceed in the same way as described above and enter further metadata for the structural element before finally creating it.

One of the main areas of work for those users with library training in the field of digitisation projects involves the comprehensive indexing of digitised objects. Basically, this covers pagination, structuring and the recording of metadata.

Those members of the digitisation project who are responsible within their user groups for recording structure data and metadata will complete their tasks in Goobi in a similar way to the other user groups described earlier (scan operators and quality control).

After logging in, they also need to click on My tasks in the menu bar and select the task they wish to process next from the list of tasks displayed. The screen containing the details of the selected task is very similar to that described above for scan operators and quality control users. Here, too, the user will find a box of general properties on the left and the Process log for general observations concerning the process. However, one additional action is available in the Possible actions area. The Edit metadata link allows the user to open Goobi’s Metadata Editor. The precise functions and user operation of the Metadata Editor are described in a separate section of this manual on account of its complexity.

Unlike project members working as scan operators or in quality control, users involved in processing structure data and metadata do not generally work directly with data from the file system. Consequently, at this point Goobi does not provide access to the digitised material in the file system within the user’s work drive. If required, however, this can also be configured for users processing structure data and metadata to allow for read-only access to the images and allowing them to be displayed using an image viewer. In this case, when the task is closed, the folder containing the digitised material would be removed from the user’s work drive to prevent any further access to that material outside the authorised workflow.

There are several ways in which you can subsequently modify the structure of a document.

One of these involves moving structure elements to a different position. To do this, first select the structure element from the structure tree. Next, in the Selected docstruct box, select the function Move docstruct up or Move docstruct down to move the selected element by one position in the required direction.

However, if you want to move the structure element to a different position within the hierarchy, you will need to select the function Move docstruct to other location. This will open a dialogue box in which you can specify exactly where you want to place the structure element. You will be able to select only from those locations permitted by the Goobi ruleset.

Within Goobi, it is also possible to copy structural elements from the METS file of a process to another process. To do this, click on the Import structural element from another process function in the Selected structural element box.

First enter the name of the process from which you want to transfer the data and then click on Search for process.

You then have the option of selecting one or more structural elements of the selected operation by clicking on them.

Once you have made your selection, click the Next button to go to the structure tree of the currently open process.

Now select the structure element into which the previously selected structure elements of the other operation are to be inserted. Please note that only those structure elements can be selected as the target that may contain the copied elements according to the rule set. Please also note in this context that this functionality only copies the logical structural elements from another process. Images and page assignments are not copied in this copy process.

This function can be used to upload a file from the user’s computer to a selected process folder.

The user will need to specify a file and choose the position within the pagination sequence where the new file is to be inserted. Next the user can choose whether this new file is to be inserted as an uncounted page or whether it should be integrated into the existing pagination. Click the Upload file button to insert the file into the currently selected process folder. If this is the media folder, the pagination sequence will be updated. The file will be inserted in the selected position and either created as a uncounted page or integrated into the existing pagination sequence. If the file is integrated, the page number for each of the following pages will increase by one. In this context, however, it is important to note that the last file in the pagination sequence is not allocated a page number. It remains unpaginated and if necessary will again have to be changed separately in the pagination sequence.

If there is already a file in the selected folder with same name as the file being updated, the file cannot be uploaded and the user will receive an error message.

Pagination is a key element of digitisation projects. It involves matching page labels (printed page numbers in the source material) to the scanned images. In older works, the page numbering often changes within the actual source text. Often, there is no printed page number, or the same page number is used more than once, and sometimes the pagination can change several times within a single work. A typical example of altered pagination occurs when the preface or introduction makes use of Roman numerals only to be replaced by Arabic numerals after the index.

To edit the pagination for the source work, open the pagination area in the Metadata Editor by clicking on the option in the navigation bar. Goobi will calculate the number of images in the current process folder and produce a vertical list. The box immediately to the right is used to allocate a pagination number to each image. As you can see below in the box entitled Page selection, Goobi has initially determined that all the pages should be numbered consecutively using Arabic numerals, beginning with the number 1.

Using either the keyboard combinations or the image navigation bar above the scanned image, you can now navigate through the entire set of images to obtain the printed page number for each image. In most cases, the first few pages of books do not contain a page number. Click the first checkbox for image 1 in the Page selection box. Next, in the Define pagination box, select unnumbered from the drop-down list and then click on the link From first selected page. For pagination purposes, you have now told Goobi to treat the volume as though it did not contain any printed page numbers. Next, move to the first image that shows a printed page number.

When you are selecting pages, please remember that you must go by the image number and not by the printed page number in the source. In Goobi, you can always tell which is the image number. Two numbers are always shown for each image, separated by a colon. The number on the left is the number of the file within the file system, i.e. the image number. The number to the right of the colon is the printed page number in the source, known as the page label. You should always take the image number (to the left of the colon) as your guide to avoid accidentally choosing the wrong pages. It is worth noting that in some source material the same printed page number (page label) can occasionally be used more than once.

Once you have identified the page where the pagination of the source material changes and marked the corresponding checkbox, choose the required pagination type in the Define pagination box. Next, enter the current page number/label and click once again on the link From first selected page. This will instruct Goobi to allocate the required page number to each image, beginning consecutively with the page you have selected.

Sometimes, the pagination of the source material can change more than once. Pages without a page number and repeated instances of the same page number occur frequently. Pages on which a new structure element begins (e.g. new chapters) but that do not contain a printed page number (although page numbering is continued on subsequent pages) are marked with a simulated pagination. This simulated pagination can be recognised in that a logical numbering is assumed despite the absence of a page number, and consequently the assumed page number is shown in brackets.

Alternatively, you can at any time click the icon next to the page number in the Page selection box to display the corresponding image in the image display area on the right.

Goobi supports different page numbering methods for pagination purposes. As well as allocating Arabic and simulated Arabic page numbers, Roman and simulated Roman page numbers, free text and unnumbered pages, you can also specify the sequence to be used for consecutive numbering. Using the symbols in the Define pagination box, you can determine how the page sequence in the book should actually appear.

Goobi supports the following page sequences:

Pagination types supported by the Metadata Editor

Setting representatives

Within the METS editor, a screen can be labelled as a representative of the object. The representative is an image that represents the work. This is usually the title page. Most digitisation portals such as Europeana, DDB, ZVDD, VD18 or the Goobi viewer use the first image of the METS file as the representative of the object, unless another image is explicitly defined as the representative.

To select the image to be used as a representative, simply click on the star icon next to the desired image. The selected image is then displayed in the form of a blue star icon. Within the METS file, the selected image is then labelled with the attribute USE="banner".

<mets:file ID="FILE_0002_PRESENTATION" MIMETYPE="image/tiff" USE="banner">
     <mets:FLocat LOCTYPE="URL"     
     xlink:href="file:///opt/digiverso/viewer/media/BV041228853/00000003.tif"/>
</mets:file>

A subfolder entitled fileupload is automatically generated on the Goobi server in the configured folder named tempfolder. This subfolder is used to exchange files between processes on the server.

In this area of the METS Editor, you can select one or more pages for export to the exchange folder on the server. A subfolder with the current process title is created in the exchange folder. Goobi will search for the selected pages in all the image directories (derivatives) for the current process and export them to the process subfolder in the exchange folder.

You can also choose to delete all the selected pages from the current process when you export them. This deletion will also be applied to the filename allocation, all allocations to structure elements and all the allocated metadata. The file will also be deleted from all the process folders.

If the currently displayed image has been deleted, Goobi will display the first image for that process. If not, the image number will simply be updated.

This section explains a number of additional functions in the METS Editor that involve directly modifying the file system for the images. For this reason you should take great care when using the functions described below.

If you wish, you can modify previously created pagination sequences. Goobi allows you to move or delete individual pages or several pages at the same time. To do this, you first need to select the required pages in the Page selection box. Once you have highlighted at least one page in this way, you will be able to choose from the following functions:

Regardless of whether or not you select a page, you can also use the following function:

If one or more pages are moved up, each page will displace its predecessor in the pagination sequence and will take over all the settings and page allocations of the predecessor. If a page is moved down, its original place it taken by the next page in the pagination sequence.

If the changes affect the image currently being displayed, the image number in the page display is updated automatically. There is no change in the displayed image.

As well as moving files, you can also delete selected files. This action deletes the selected page completely from the metadata file. It affects both filename allocation and allocation to structure elements or allocated metadata.

The file is also deleted from all the process folders. You can search for files that you wish to delete from all the process folders using the selected filename. Goobi will search the available folders for files that match the filename (but not necessarily the file extension). The file will be deleted in all the subfolders of the ocr and images directories. If the currently displayed image has been deleted, Goobi will display the first image. If not, the image number will be updated.

Goobi also allows you to generate new filenames so that systems that are not based on the sequence of filenames in the METS file can continue to display the images in the correct sequence. All the files are numbered using an eight-digit system based on the sequence in the METS file. The files will be renamed in all the process folders.

In the following sections you will get a detailed overview of several management areas, as for example for the creation of workflows, management of projects, users, user groups, statistics and rulesets.

There are various keyboard combinations available in the metadata editor. These are summarised here once again:

This function can be used to download a file from the current process folder to the user’s own computer. The user can select the file to be stored locally using the computer’s file browser.

Provided that the server-based target folder is not empty, you can use the Server-based import box to select a folder from which previously exported images can now be imported into the current process.

In order to do this, you will need to specify the point in the pagination sequence where you want to insert the new file. You can then choose whether the new files are to be inserted as uncounted pages or whether they are to be integrated into the existing pagination.

For every subfolder in the target folder, Goobi will then search for a counterpart in the current process folder. The selected files are then imported to this folder. Once the files have been successfully imported, the subfolder in the target folder is deleted.

The metadata editor has an editor for OCR results with which the ALTO XML files in the process can be edited. This editor can be accessed via a button in the upper menu bar:

The ALTO editor overlays the usual interface of the metadata editor. The interface consists of two main elements. The digital copy is displayed on the left and the automatically recognised OCR text on the right. If you move the mouse over the image or over the text on the right, the recognised line and the recognised word are highlighted in the image:

The editor allows you to correct the words recognised by the OCR. To do this, simply click in the text on the right and change the words as you are used to doing with text editors. When you edit a word, it is also marked on the left of the digital copy. If a word is difficult to read, you can zoom in the image display with the mouse wheel. In addition, the page to be edited can be selected via the navigation below the image.

To save the results, just click the green button at the bottom right. To undo all changes, the editor can simply be closed.

In order to specify and manage the users that you want to be able to work with your Goobi installation, first select the menu item Administration - Users from the menu. This will display a list of all the users currently entered in Goobi. In the Users column, you will see the first name and surname of each registered user. The Location column tells you the establishment or town where the user is based. The User groups column shows you the user groups to which that user belongs. In the Projects column, you can see which projects each user has been assigned to.

Finally, in the Actions column, you can edit the details for each user. As an administrator, you now have an option to select a user and switch to that users role.

To do so, click on the icon to the right in the Actions column. This will switch your own authorisation level and role as an administrator to those of the selected user, allowing you for test purposes to check how Goobi behaves for a specific user. This can be particularly useful if individual users have reported problems that cannot otherwise be traced. By switching to the authorisation level and role of a specific user, you can see exactly how the interface appears to that user without that person having to give you his/her user name and password.

Just above the list of users, you will find an option that allows you to display inactive users as well as active users. To do so, click on the checkbox Only show active users. This will remove the tick and deactivate the default setting which restricts the display to active users.

You can also use the filter above the table to search for a specific user from a large list. Simply enter part of that user’s first name or surname and press the enter key to conduct the search.

To add a new user, click on the Create new user link below the table. To edit the details for an existing user, click on the first icon in the Actions column for that user.

The Edit user dialogue box allows you to change the details already stored in Goobi for that person. As well as the Surname, First name and Location, you can also assign a Goobi Login and Password.

If Goobi has been instructed at the main configuration stage to authenticate each user against a configured LDAP, you will at this point need to specify for each person which server is to be used for authentication.

In the Metadata language input field, you can use a language code in free text form. Please ensure that the language code you use matches exactly the code defined in the ruleset with which users will be working in the METS Editor. In the example given below, the rulesets must contain a localised form of the metadata and structure data descriptors (language) for English with the language code en. If the field is left empty, the language set in the browser is used in the metadata editor.

Localisation of the metadata item 'logicalPageNumber' for various languages within a ruleset

<MetadataType>
    <Name>logicalPageNumber</Name>
    <language name="de">logische Seitenzahl (gedruckte Seitenzahl)</language>
    <language name="en">logical page number</language>
    <language name="es">número de página lógico</language>
</MetadataType>

You can specify in the Edit user dialogue box whether the user’s Goobi account should be active or inactive. This means you can configure Goobi in such a way that existing user accounts remain in place even though the corresponding users are no longer authorised to access those accounts. This is particularly important to avoid any potential loss of existing information from source material that has already been processed or any difficulties conducting statistical searches that might otherwise be caused by the removal of individual users from the database. It also ensures that information about source material whose workflow was completed some time ago can be analysed precisely and that project managers or administrators can identify which person performed which tasks and when. This can be particularly useful in the event of an error.

By activating the batch download checkbox, you can allow individual users to display a more extensive list of tasks of possible actions underneath the My tasks list. These enable the user to accept several tasks at the same time and to work on them all together, e.g. in an automated batch process to optimise image quality. Once tasks such as these are completed, the source work images corresponding to the process will be stored in the folder provided by Goobi for batch upload on that user’s work drive. This folder is often called 'Ready', 'Done' or similar. This can be configured individually for each Goobi installation. Placing more than one folder of related tasks in the batch upload folder allows users to close all finished tasks for which there is a related folder in the batch upload folder by means of a single click in the Possible actions area. This method enables users to accept and close a large number of tasks simultaneously.

Thanks to its comprehensive system of user authorisations, Goobi ensures that each person can be assigned not only to different projects but also to different roles. Furthermore, every user can perform more than one role at the same time. In this context, roles can also be understood as levels of qualification or as skills. These roles can be assigned in the User groups area. To assign additional roles to a specific user, simply click on the Add user groups button for that user.

In the User group window, select all the user groups to which you want to assign the user and then click on the Save button to confirm your selection. You will find that the selected user has now been assigned to those user groups. You can follow the same process to assign users to projects.

To remove a person from a particular user group or project, click on the symbol for deletion next to that user group or project. Next, to apply permanently any changes you have made, click on the Save button.

update benutzer set superadmin = true where BenutzerID=123456;

Goobi users with a higher level of authorisation will find a number of additional options in the menu bar. These allow you to amend data that may relate to an entire project or to different users and user groups. Such changes can only be made by users with the appropriate level of authorisation.

One of Goobi’s key features is the integration of both Goobi and the additional services linked to it to a central authentication system. In most cases, the authentication process is conducted via a central LDAP server that allows users to work in different systems with a single user account and just one login and one password.

This means it is possible to log in to Goobi with your own user account and to access the work drive on which Goobi provides the data to be processed by users.

Access to the work drive is provided in the background via a Samba server that authenticates user requests via the central LDAP servers. To configure the LDAP groups, you need to click on the menu item Administration - LDAP groups. Goobi will display an overview of all the LDAP groups that have been set up. In most cases, only one LDAP group will be configured.

Klicken Sie auf das Icon zur Bearbeitung einer der konfigurierten LDAP-Gruppen, so gelangen Sie in die Bearbeitungsmaske. Innerhalb dieser können Sie die einzelnen Parameter für die LDAP- Authentifizierung konfigurieren.

After changing the LDAP group parameters, click on the Save button to return to the list of LDAP groups and apply your changes. Clicking on the Delete button will delete the selected LDAP group permanently from Goobi. You should note that users will then no longer be able to access the authentication function. It is not possible to continue working with Goobi if an LDAP group has been incorrectly configured or deleted.

In order to ensure flexibility in the handling of freely configurable workflows and the corresponding users, you will need to define a number of user groups. These user groups can be understood as the roles, activities or skills of individual users within the workflow. By way of example, user groups can be created for scan operators, metadata editors and administrators. You are free to define these groups as you wish in Goobi and then assign individual users to them. To open the list of existing user groups, click on the menu item Administration - User groups in the menu. Goobi will display a list of all the user groups that have already been defined. For each of these user groups, you can also display a list of all the users assigned to that group. To show this list, simply click on the little symbol to the left of the user group name.

To create additional user groups, click on the Create new user group link. To edit existing user groups, however, you need to click on the Edit symbol in the Actions column.

Within the editing mask, a title must always be assigned to a user group. This can be freely selected. Please make sure that you select this title as descriptively as possible to give you and other administrators in Goobi a better overview.

Very granular permissions can now be assigned to each defined user group. These can be added in the right area of the form by simply clicking on an authorization. A click on an authorization in the left area removes it from the user group.

The following authorizations are available for selection:

Authorizations at administrative level

Administration menu

This permission allows the user to view and use menu items in the Administration menu.

Administrative Tasks

This permission allows access to the administrative area, in which a message can be sent to all logged in users.

Dockets

This authorization allows you to list and edit dockets.

LDAP groups

This permission allows you to list and edit LDAP groups.

Administration plugins

This authorization allows the execution of administration plug-ins, as long as they are installed and do not require special individual rights.

Projects

This authorization allows you to list and edit projects.

Rulesets

This authorization allows you to list and edit rulesets.

User Groups

This authorization allows you to list and edit user groups and their authorizations.

User

This authorization allows you to list and edit users.

Allow user switch

This authorisation allows you to switch to the user account of another Goobi user without having to use their login data. A precondition for this is, however, that the User authorization and the Administration menu are also available.

Show current users

This authorization allows you to display information on how many users are currently actively working in Goobi.

See current users details

This authorization allows you to display the number of active users in Goobi as well as a detailed list of exactly which users are involved and when they last showed activity.

Delete files from process log

With this permission the user has the possibility to delete files that have been uploaded for a process within the process log or in one of the directories import/, export/ or source/.

Notifications for all status changes

Users with this permission can configure for which status changes they want to be notified. Within the user settings of each user, there are fields available that define the notification for status changes by mail. A notification is possible for the status change of tasks in the following cases:

• When tasks are ready for editing

• When tasks are accepted for processing by a user

• When tasks have been completed

• When tasks reach an error status

Authorizations for Statistics

Statistics menu

This permission allows the user to view statistics and use menu items in the menu.

General Statistics

This authorization guarantees access to the standard statistics available within Goobi.

Statistics plugins

This authorization allows additionally installed statistics plug-ins to be called.

Authorizations for Tasks

Task menu

This permission allows the user of the My Tasks menu to view and work with My Tasks.

Task List

This permission allows you to view and work with the user's own task list.

METS Editor - Pagination

This authorization determines whether users with access to the internal metadata editor are allowed to work with the Pagination area.

METS Editor - Structural Data

This authorization determines whether users with access to the internal metadata editor are allowed to work with the Structure Data area.

METS Editor - Metadata

This authorization determines whether users with access to the internal metadata editor are allowed to work with the Metadata area.

METS Editor - Files

This authorization determines whether users with access to the internal metadata editor are allowed to work with the Files area.

Authorizations for Workflows

Workflow menu

This permission allows the user to see and use menu items in the Workflow menu.

Search for processes

This authorization controls whether the user may use the search mask to search for processes.

Show processes from other projects

This authorization determines whether a user may also see processes in which he himself is not a member.

Batches

This authorization determines whether the user may group processes into batches or modify existing batches.

View workflow details

This permission determines whether a user is allowed to see the details of a process.

Edit workflow details

This permission determines whether a user can see and edit the details of a process.

Workflow plugins

This authorization allows additionally installed workflow plug-ins to be called.

Authorizations for Production Templates

List of process templates

This authorization controls whether a user has access to the list of process templates.

Clone process templates

This authorization determines whether a user may clone an existing process template in order to create a new process template.

Creating process templates

This authorization determines whether a user can create new process templates.

Single imports

This authorization enables the user to create new processes individually on the basis of existing process templates.

Mass imports

This authorization enables the user to create new operations on the basis of existing process templates by carrying out a mass import.

Authorizations for operations

List processes

With this authorization, the user has the option of having processes listed.

Change process template later on

With this authorization, the user has the option of subsequently changing the process template to be used for an existing process.

Download of process information

With this authorization, the user has the option of downloading process information as an XML file.

Export of processes

With this authorization the user is able to perform a tabular export of a process list.

Linking of processes in home directory

This authorization can be used to determine whether a user may create a link to the data in his home directory for a desired process.

Display deactivated projects

This permission can be used to determine whether a user can also see processes from deactivated projects.

Display finished processes

This authorization can be used to specify whether a user is to be given access to finished processes.

Run all GoobiScripts

This permission gives the user the rights to apply all GoobiScripts over a number of operations.

Individual GoobiScripts

It is possible for user groups to be given specific rights to execute selected GoobiScripts. For this purpose, the assignment of the permission is not done by clicking on one of the indicated rights in the right area. Instead, the GoobiScript to be allowed is named in the right input field and is accepted by clicking on the button with the plus icon. The assignment of the permission is done in such a way that goobiscript_ is entered followed by the name of the concrete GoobiScript.

Accordingly, the following authorizations can be assigned as examples:

One of the fundamental aspects of Goobi is that all structure data and metadata to be recorded as part of a digitisation project can be specified on the basis of a flexible system of rulesets. Before the project actually begins, the project management team can decide which structure elements and which metadata elements should be admitted for that project. The collection of these structure data and metadata, the user authorisations governing how and how often they may be used and the associated translations for the interfaces are specified in the ruleset files.

A detailed description of how to create a ruleset file can be found in a separate document. This documentation can be found here:

This section explains how you can work with rulesets in Goobi. To do this, click the menu item Administration in the top menu bar and choose Rulesets from the drop-down menu. This action will display a list of the rulesets that have already been defined.

If you click on the Create new ruleset link, Goobi will display a dialog box allowing you to create an additional ruleset.

You will need to enter a meaningful name that you wish to use when allocating project tasks. For the ruleset file, you must choose the actual name it has been given in the file system.

In the ruleset dialogue box, as well as choosing the title and file name of the ruleset, you can stipulate the order in which data from that ruleset are displayed in Goobi. The default setting is alphabetical. This means that all lists of structure element names and metadata fields will be sorted in alphabetical order in the Goobi METS Editor. However, if instead you activate the checkbox Sort as in ruleset in the dialogue box, the order will change. For the configured ruleset in question, the structure data and metadata will now be shown in the order they appear within the ruleset. This can be useful, for example, if you specifically want to make a manual change to the order in which Goobi displays the structure data and metadata. To save the changes you have made in the dialogue box, simply click the Save button. To delete the selected ruleset, click on the Delete button. When entering the file name, please ensure that the specified file actually exists in the configured ruleset. The default path that Goobi expects you to use when saving your rulesets is:

/opt/digiverso/goobi/rulesets/

In the figure above, for example, the file archive.xml must exist in the rulesets folder in order for Goobi to use it:

/opt/digiverso/goobi/rulesets/archive.xml

If you want to search for specific processes, you can use Goobi’s extended search function. To do this, select the menu bar option Workflow – Search for volume.

In order to conduct a finely tuned search, Goobi’s Search for volume dialogue box uses a filter syntax that makes it possible to identify the requested processes on the database.

The above diagrams show how a search is performed in Goobi using its internal search syntax based on the search details entered in the dialogue box. The search syntax generated by Goobi can be seen in the filter box Find processes above the table and can be extended to include additional parameters. It is also possible to save a search in case it is needed on future occasions. It will then appear in the drop-down list of pre-defined filters and can be executed at any time.

Below you will find an explanation of the background filter syntax. This is useful because manually combining the search parameters described below allows you to perform more detailed searches than is the case simply using the dialogue box. The following table contains a description of some typical search filters and how they work.

Examples of the detailed filter syntax for use with process searches

Note: The operators < and > are interpreted in the background as less than or equal to and greater than or equal to. For example, searching for processdate>2022 will list all processes whose creation date is 2022 and later.

As you can see from the examples given above, it is possible to conduct some very complex filter-based searches by combining a wide range of parameters. Unlike the simple filters you can apply in the dialogue box, by using the correct filter syntax you can use the same parameters more than once with different values for a single search (e.g. to simultaneously retrieve both completed workflow steps and others that are still in progress).

All the parameters listed above can be combined with each other as required. Please note, however, that any parameters containing a blank space should be written inside double quotation marks.

If you need even more detailed information about a process, click on the first symbol in the Actions column for that process. This will open the detailed view window for that process with a number of options for specific purposes. You can also perform other actions for each process. Details of these possible actions can be found in the Actions column for each process. Some of the possible action symbols will not be activated if the status of the process does not yet permit that action to be performed. You cannot, for example, generate a pdf for a process if it does not yet contain any images. In such cases, the Generate PDF symbol will be deactivated.

The following options are available for each process in the Actions column:

Possible actions for individual processes

It is important to note that all of the possible actions offered by Goobi can be performed for individual processes or a number of processes together regardless of their current status. This allows users with manager-level or administrator-level authorisation to access and if necessary amend the data for specific processes at any time.

As well as the options described above allowing users to edit or amend each volume independently of the current status of the workflow and to process its structure data or metadata or re-export the data to the presentation system, with Goobi you can also apply different actions to a whole group of processes. These actions are applied to all of the processes displayed in the table. If you want to restrict the change or action to a particular selection of processes, you will need to filter the list accordingly. To do so, simply use the Filter processes box, the list of predefined filters or Goobi’s search function to list only those processes you wish to edit.

Once you have adjusted the list of processes to meet your requirements, you can apply the available actions (shown underneath the process table) to that list. For each action, you can also choose whether to apply the action to the entire set of matches (i.e. all the filtered processes) or just to the processes listed on the page of the table that is currently being displayed. You will be prompted to make this selection when you click on one of the available actions.

The table below contains a description of each possible action.

Description of actions that can be applied to a group of processes

When you apply an action to a group of processes, you also have an option to run Goobi scripts within that action. To do so, click on the button Execute GoobiScript in the Possible actions box. Goobi will display an overview of all the scripts that can be applied to the entire list of processes, to the processes listed on the current page or just to a selection of processes.

For every GoobiScript, you need to enter the name of the script you wish to run as well as the corresponding parameters. These parameters are shown when you click on the script in the list. Replace the parameters shown as examples with your desired settings.

After completing the GoobiScript, you can now apply it to selected hits or the entire hit list. Before the execution, however, a security query appears in which the number of operations to which the GoobiScript should be applied must first be confirmed again.

Syntax

---
# This GoobiScript allows to add a new workflow step into the workflow.
action: addStep

# Title of the workflow step to add
steptitle: Upload images

# This number defines where in the workflow this new step is ordered into.
order: 3

The lines starting with a hash # form comments for help purposes. They can also be omitted before submitting the GoobiScript. Accordingly, the GoobiScript is a bit more compact afterwards:

---
action: addStep
steptitle: Upload images
order: 3

Run multiple GoobiScripts together

Thanks to the change to the new syntax, it is now also relatively unproblematic to have several GoobiScripts started together. Note that each GoobiScript is separated from the previous GoobiScript by the --- character. This way you can easily combine several commands and start them together. This could look like this, for example:

---
# This GoobiScript allows to add a new workflow step into the workflow.
action: addStep

# Title of the workflow step to add
steptitle: Upload images

# This number defines where in the workflow this new step is ordered into.
order: 3

---
# This GoobiScript allows to assign a user group to an existing workflow step.
action: addUserGroup

# Title of the workflow step to be edited
steptitle: Upload images

# Use the name of the user group to be assigned to the selected workflow step.
group: Photographers

---
# This GoobiScript allows to change the current status of a specific step in the workflow.
action: setStepStatus

# Title of the workflow step to be changed
steptitle: Upload images

# Value of the status. Possible values are `0` (locked), `1` (open), `2` (in work), `3` (done), `4` (error), `5` (deactivated)
status: 1

---
# This GoobiScript allows to add a plugin to a defined workflow step
action: addPluginToStep

# Title of the step to adapt
steptitle: Upload images

# Name of the plugin to be assigned to the workflow step
plugin: intranda_step_fileUpload

Here, too, the commas can be omitted accordingly for a shortened application, so that the entire call becomes clearer:

---
action: addStep
steptitle: Upload images
orderorder: 3
---
action: addUserGroup
steptitle: Upload images
group: Photographers
---
action: setStepStatus
steptitle: Upload images
status: 1
---
action: addPluginToStep
steptitle: Upload images
plugin: intranda_step_fileUpload

The processing of such multiple GoobiScripts takes place after sending in the order of the naming in each case over all affected processes. Related to this example this means, if 3 processes are concerned the following processing:

Available GoobiScripts

You can choose from the following Goobi scripts:

GoobiScript: addUser

The GoobiScript addUser allows you to add a new user to a specific workflow step. Before you apply this Goobi script, you should ensure that you have the correct login name for the user you wish to add to that step. You also need to check the exact name of the step to which you want to add a new user. For the parameter steptitle you should select the name of the step to which you want to add the new user.

GoobiScript: addUserGroup

The GoobiScript addUserGroup is similar to addUser, as it gives additional user rights for workflow steps. For the parameter steptitle, enter the full name of the step to which you want to add a user group, and for the parameter entitled group enter the exact name of the user group you wish to add for that step.

GoobiScript: cloneProcess

The GoobiScript cloneProcess allows the duplication of one or more Goobi processes. The parameter content can be used to specify whether only the database contents and the METS file should be copied or whether all the associated directories (e.g. the images) should also be duplicated. The parameter title can be used to control the titles of the processes to be created. The variable system of Goobi is used here and thus allows a high degree of flexibility.

GoobiScript: renameProcess

The GoobiScript renameProcess allows you to rename a process or several Goobi processes. A search string is defined with the search parameter, the new value with replace and the search method with type (defined values contains - the search value should be contained in the process title - or full - the process title must match the search value exactly). Both the task title in the workflow and the image directories of the process in the file system are renamed.

GoobiScript: deleteTiffHeaderFile

For the GoobiScript deleteTiffHeaderFile there is no need to enter additional parameters. Running this GoobiScript will delete any previously created TIFF header files that can be used by a program that writes the TIFF headers into the images. This allows you, for example, to make centrally modified TIFF headers available for future use, since missing TIFF header files are automatically created on the basis of the configuration the next time the file is accessed.

GoobiScript: swapSteps

The GoobiScript swapSteps allows you to swap the order of two steps within the workflow of a number of processes. To perform a swap, you need to provide the details of each of the steps involved. Enter the workflow number and full name of the first and second step. Running this script will then swap the order of the steps you have specified. This makes it very easy to change workflows across a large number of processes.

GoobiScript: importFromFileSystem

The GoobiScript importFromFileSystem imports existing image sets from a defined output directory into processes that have already been created in Goobi. This can be useful if you want to import projects into Goobi that were created before Goobi was installed. Please note that all the image directories within the specified output directory must have the same name as the processes in Goobi. An automatic import from the file system can only be performed correctly if the folder name and process title are identical. For the parameter entitled sourcefolder, you need to specify the location of the individual directories containing the processes you wish to import.

GoobiScript: setRuleset

The GoobiScript setRuleset allows you to make a central change to the Goobi ruleset for a group of processes. This could be particularly important after detailed editing and testing of a ruleset (for safety reasons this is performed separately in a newly created ruleset), if you then wish to apply the new ruleset to the processes. For the parameter entitled ruleset, you need to specify the name of the ruleset using the name as it appears in the ruleset list in Goobi. The newly assigned ruleset will be entered when you run the GoobiScript, regardless of which ruleset is currently in place for the individual processes being changed.

GoobiScript: deleteStep

You can run the GoobiScript deleteStep if you want to delete a specific step from the workflow for a group of processes. Running the script will delete the workflow step (specified by its full name in the parameter steptitle) from the list of selected processes. Please note that this GoobiScript will also delete any production-related data being stored for that particular workflow step (e.g. project staff, processing date, status).

GoobiScript: addStep

The GoobiScript entitled addStep allows you to automatically create a new step with a specific name and a specific position in the workflow order. For the parameter steptitle, enter the name of the new step, and for the parameter order enter the required workflow order number. In addition to numerical values, the keyword end can also be used to add the step to the end.

GoobiScript: addStepAtOtherStepPosition

The GoobiScript addStepAtOtherStepPosition enables the creation of a workflow step with a defined title at a defined position within the workflow where another workflow step is already located. By inserting the new workflow step, all existing workflow steps with this or a subsequent position are moved so that the new workflow step can be inserted at the desired target position. The parameter newsteptitle allows you to define the title for the new workflow step to be inserted. The parameter existingsteptitle defines the name of the workflow step that determines the target position of the step to be inserted. The parameter insertionstrategy defines whether the new step is to be inserted before (before) or after (after) the specified existing step.

GoobiScript: setStepStatus

You can choose the GoobiScript setStepStatus to modify the workflow status for a group of processes at the same time. For the parameter steptitle, you need to enter the name of the workflow step whose status you wish to change. For the status parameter, you should enter the required numerical value using the system:

0 = locked
1 = open
2 = in progress
3 = closed
4 = error
5 = deactivated

GoobiScript: setStepNumber

Using the GoobiScript setStepNumber you can modify the workflow order number of an individual step for a group of processes. For the parameter steptitle you need to enter the full name of the workflow step you wish to change. For the number parameter you should enter the workflow order number you want to apply to that step for all the selected processes.

GoobiScript: addShellScriptToStep

The GoobiScript addShellScriptToStep allows you to add shell scripts or other command-line calls to designated workflow steps in a group of processes. For the parameter steptitle you need to specify the full name of the steps you wish to change. For the script parameter, enter the full command that you wish Goobi to execute in the form of a command-line call whenever this step is activated.

Please note that shell commands at Linux level begin with /bin/bash/.

In the parameter label you define the name for the shell script.

GoobiScript: setStepProperty

You can use the Goobi script setStepProperty to set individual options for a specific workflow step in a group of processes at the same time. For the parameter steptitle, you should enter the full name of the step you wish to select. For the property parameter, you will need to select one of the following values:

You should also set the value of the actions you have specified here to activated or deactivated by entering the values true or false for the value parameter..

GoobiScript: export

The GoobiScript export allows you to export a large number of processes. The parameters exportImages and exportOcr can be used to specify whether the associated images and OCR data should be exported. If an export plugin has been configured in the workflow, that plugin will be loaded and used for the export; if not, Goobi will run the default export.

GoobiScript: runScript

Using the GoobiScript runScript, you can initiate a script for a particular workflow step outside the regular workflow. The parameter steptitle is used to enter the full title of the workflow step whose scripts you wish to run.

If the workflow step contains a number of scripts, you can specify which one you wish to run using the script parameter. If this parameter is left blank, all the scripts for that workflow step will be run in the specified sequence.

GoobiScript: deleteProcess

As the name suggests, the GoobiScript deleteProcess is used to delete processes. You have to use the parameter contentOnly (value true or false) to specify whether Goobi should delete only the data from the file system or, additionally, all the information from the database.

GoobiScript: addPluginToStep

The GoobiScript addPluginToStep allows you to add plugins to workflow steps. You can use the parameter steptitle to specify the name of the workflow step and the parameter plugin for the identifier of the plugin that you wish to add.

GoobiScript: updateImagePath

The GoobiScript updateImagePath updates the path to the image files within the METS files. No parameters are required to run this GoobiScript.

GoobiScript: updateContentFiles

The GoobiScript updateContentFiles updates the list of all image files within the METS files. No parameters are required to run this GoobiScript.

GoobiScript: addToProcessLog

The GoobiScript addToProcessLog allows adding messages to the process log. The type parameter determines how the message should be classified. The message parameter specifies the content of the message.

GoobiScript: setProject

The GoobiScript setProject allows you to assign the selected tasks to a defined project. The parameter project specifies which project should be used for this.

GoobiScript: runPlugin

The GoobiScript runPlugin allows the execution of a step plugin for the selected tasks. The parameter steptitle determines the step of the affected tasks from which the plugin is to be executed.

GoobiScript: import

The GoobiScript import is not intended for execution by users from the user interface. Instead, it is started during the execution of mass imports from the selected plugin. It then performs a mass import in the form specified in the import plug-in. The parameter plugin defines the unique name of the plugin. The identifiers parameter determines which identifiers the data records have that are to be imported. The parameter template determines which production template is to be used for the import.

GoobiScript: metadataDelete

The GoobiScript metadataDelete allows you to delete metadata from a process. The field parameter specifies the type of metadata, where the internal rule set name must be used. The value parameter defines the content of the metadata. The parameter ignoreValue determines whether the content of the parameter value is to be ignored and whether the metadata is to be deleted independently of its value. The parameter type can be used to control whether the metadata to be deleted are present as normal metadata, whether they occur within a named metadata group, or whether an entire metadata group is to be deleted. The parameter group allows the naming of the group concerned. The following application scenarios apply:

• If metadata is given as value within type (or no value is given) and no name is given within the parameter group, a normal metadata is deleted.

• If metadata is specified as the value within type (or no value is specified) and a name is specified within the parameter group, the metadatum is changed within the named group.

• If group is specified as the value within type, the group named within the parameter field is deleted.

The parameter position allows you to specify where the metadata should be found:

---
action: metadataDelete
field: DocLanguage
value: deutsch
position: child
ignoreValue: false

---
action: metadataDelete
field: DocLanguage
value: deutsch
position: top
ignoreValue: true

GoobiScript: metadataAdd

The GoobiScript metadataAdd allows you to add new metadata to a process. The field parameter defines the type of metadata, where the internal ruleset name must be used. The value parameter defines what content the new metadata should contain. The parameter ignoreErrors determines whether, in the event of an error, the processing and saving of the METS file should be continued or the processing for the object should be aborted. The parameter type can be used to control whether the change is to be made to a simple metadata or within a metadata group. The parameter group defines the metadata group to which the metadata is to be added. The following application scenarios exist:

• If metadata is given as value within type (or no value is given) and no name is given within the parameter group, the metadata is added as normal metadata.

• If metadata is given as the value (or no value is given) and a name is given within the group parameter, a metadata is added to the named group.

• If group is specified as the value, a new metadata group is created with the name specified within the group parameter.

The parameter position allows you to specify where the metadata should be added:

Any authority data can also be added using the authorityName and authorityValue parameters.

---
action: metadataAdd
field: DocLanguage
value: deutschTop
position: top
ignoreErrors: false

---
action: metadataAdd
field: DocLanguage
value: deutschChild
position: child
ignoreErrors: false

---
action: metadataAdd
field: DocLanguage
value: deutschChild
position: child
ignoreErrors: false
authorityName: gnd
authorityValue: 123456789X

GoobiScript: metadataReplace

The GoobiScript metadataReplace allows you to replace a metadata with a new value. The old value is thus replaced by another value and is therefore no longer available. The field parameter determines which type the metadata has, whereby the internal ruleset name must be used here. The search parameter defines the current content of the metadata. The replace parameter defines which content the metadata is to have instead. The parameter group specifies whether the change is to be made within a metadata group. If no value is given here, the change is made to a normal metadata. If, on the other hand, a value is given, the change of the metadata takes place within the named metadata group. The parameter position allows you to specify where the metadata to be replaced should occur and be replaced:

Any authority data can also be added or changed using the authorityName and authorityValue parameters.

---
action: metadataReplace
field: DocLanguage
search: deutschTop
replace: deutschNewTop
position: top

---
action: metadataReplace
field: DocLanguage
search: deutschChild
replace: deutschNewChild
position: child

GoobiScript: metadataReplaceAdvanced

The GoobiScript metadataReplaceAdvanced allows replacing a metadata with a new value. In contrast to metadataReplace, regular expressions can be used here to manipulate values. The field parameter determines what type the metadata has, whereby the internal ruleset name must be used here. The value parameter defines a regular expression that is applied to the content of the metadata. The parameter group specifies whether the change is to be made within a metadata group. If no value is given here, the change is made to a normal metadata. If, on the other hand, a value is given, the change of the metadata takes place within the named metadata group. Any authority data can be added or changed using the authorityName and authorityValue parameters.

---
action: metadataReplaceAdvanced
field: DocLanguage
value: s/deutsch/german/g
position: top

GoobiScript: metadataChangeValue

The GoobiScript metadataChangeValue allows the manipulation of existing metadata of a process. Prefixes or suffixes can be added to an existing metadata to extend the content of a metadata. The field parameter specifies the type of metadata, where the internal ruleset name must be used. The content of the prefix parameter is used to prefix a text with the current value of the metadata. The content of the parameter suffix is used to append a text after the current value of the metadata. The parameter group specifies whether the change is to be made within a metadata group. If no value is given here, the change is made to a normal metadata. If, on the other hand, a value is given, the change of the metadata takes place within the named metadata group. The parameter position allows you to specify where the metadata should be present and adjusted:

---
action: metadataChangeValue
field: DocLanguage
prefix: start_
suffix: _end
position: top

---
action: metadataChangeValue
field: DocLanguage
suffix: ist eine schwierige Sprache
position: top
condition: Deutsch

---
action: metadataChangeValue
field: DocLanguage
prefix: start_
suffix: _end
position: child

GoobiScript: metadataChangePersonType

The GoobiScript metadataChangePersonType changes the role type of a person. The call requires four parameters. The oldType parameter specifies what the old type should be, and the newType parameter specifies the type that the person should receive instead. The parameter ignoreErrors determines whether, in the event of an error, processing and saving of the METS file should continue or whether processing should be aborted for the object. The parameter position, on the other hand, controls where the person should be present to be changed:

GoobiScript: metadataChangeType

The GoobiScript metadataChangeType changes the type of a metadata. The call requires four parameters. The oldType parameter specifies the old type of the metadatum. The newType parameter specifies the new type for the metadata. The parameter ignoreErrors controls whether, in case of an error, the processing and saving of the METS file should be continued or whether the processing for the object should be aborted. The parameter type can be used to control whether the change is to be made to a simple metadata or within a metadata group. The parameter group defines the metadata group for which the metadatum is to be changed. The following application scenarios exist:

• If metadata is specified as value within type (or no value is specified) and no name is specified within the parameter group, a normal metadata is changed.

• If metadata is specified as the value within type (or no value is specified) and a name is specified within the parameter group, the metadata is changed within the named group.

• If group is specified as the value within type, the type of the named group is changed from the old value (oldType) to the new value (newType).

The parameter position allows you to specify where the metadatum should occur so that it is affected by the change:

GoobiScript: changeProcessTemplate

The GoobiScript changeProcessTemplate allows you to change the workflow for the affected processes. The templateName parameter defines which production template is to apply to the operations. If this GoobiScript is applied to processes, Goobi tries to set the steps already performed to the identical status in the updated workflow if possible. This can only succeed if the steps have the same titles.

GoobiScript: updateDatabaseCache

The GoobiScript updateDatabaseCache ensures that the internal database table of the Goobi database is updated with the status of the workflows and the associated media files as well as metadata. This is important if, for example, the metadata has been modified outside of Goobi, or if a new index field has been defined. Among other things, various statistics are based on these database tables and therefore require as up-to-date values as possible for the visualization of information.

No parameters are required to run this GoobiScript.

GoobiScript: propertySet

The GoobiScript propertySet allows you to add and change a process property. The parameter name specifies the name of the property. The value parameter specifies the value that the property should have. If a property already exists with the name specified here, its value is changed to the value specified here.

GoobiScript: propertyDelete

The GoobiScript propertyDelete allows the deletion of process properties. The parameter name specifies the name of the properties to be deleted.

GoobiScript: executeStepAndUpdateStatus

The GoobiScript executeStepAndUpdateStatus executes a selected step and then updates the workflow for further processing of the following work steps. The steptitle parameter determines which step is to be executed. After the call, Goobi checks whether this is a script step, an export step, a plug-in step or an HTTP step and executes it accordingly. If the work step is marked as automatic, the further workflow process is continued after the execution. If, on the other hand, the call triggers a wait mode, the status is not changed by Goobi but waits for a status change by the respective plugin or script itself. If an error occurs while the started step is being executed, the status of the workflow step is set to error.

GoobiScript: exportDatabaseInformation

The GoobiScript exportDatabaseInformation exports all database contents of the selected Goobi operations to an internal XML file. This is then located in the Goobi file system within the process folder and has a file name that can be used to import the data into another Goobi instance. The path of such a file is e.g.

/opt/digiverso/goobi/metadata/123/123_db_export.xml

No parameters are required to run this GoobiScript.

GoobiScript: moveWorkflowForward

With the GoobiScript moveWorkflowForward the status of the workflow can be moved forward step by step. Each time this GoobiScript is executed, the active workflow step is changed accordingly (e.g. from open to in work).

GoobiScript: moveWorkflowBackward

With the GoobiScript moveWorkflowBackward the status of the workflow can be moved backwards step by step. Each time this GoobiScript is executed, the active work step is changed accordingly (e.g. from completed to in work).

GoobiScript: setPriority

The GoobiScript setPriority can be used to define the priority of individual or all process steps. The parameter priority determines which priority should be used. The following values are available: standard, high, higher, highest and correction. The parameter steptitle determines for which workflow step the priority is to be set. If the parameter steptitle is not specified, the priority is changed for all workflow steps of the selected processes.

As a workflow management application for the library environment, Goobi has to be able to deal with a wide range of specific configurations and project-specific requirements. To this end, it has been designed in line with established conventions. These cover individual directory structures and the way Goobi uses these structures in different areas of the application. This section outlines the directory structures that have proven most effective and explains how external storage is integrated into the system.

All objects that are processed in Goobi as part of the whole range of digitisation workflows, and that as such have their own individual progress status, can be viewed in the Processes area. First, click on the menu item Workflow – Processes in the menu. Goobi will now display a list of all the processes you are authorised to view. If you are a Goobi administrator (as in the screenshot below), you are authorised to view all the processes being managed in Goobi.

If you are not an administrator but are authorised to Goobi manager level, you will have been assigned to specific projects. In Process view, you will only be able to display those processes you are authorised to access through your membership of those projects.

The number of hits listed in the table of processes is determined by each individual’s user configuration. In the diagram shown above, for example, you can see the number of rows in the table is limited to ten. In order to display more processes, you would need to move to the next page using the page function below the table.

You can use the simple checkbox filters above the table to quickly filter the results displayed by Goobi or to show processes that are hidden by default. By way of example, an administrator can select the checkbox Show deactivated projects above the process table to display all the processes that were previously hidden because the corresponding project has been deactivated. Using the checkbox Show finished processes, you can display all the processes that belong to active projects but have already completed every step in their workflow. By default, if neither of these checkboxes is selected, Goobi will only show processes that are currently being processed as part of the workflow, i.e. completed processes and deactivated projects are excluded by default.

After using a particular filter, if you wish to save it for future use, you can store it in the list of pre-defined filters. To do so, once you have entered the filter string, simply click on the save symbol next to the drop-down list of pre-defined filters. If you wish to reuse any of your pre-defined filters on a future occasion, simply choose it from the drop-down list to enter the filter string automatically in the Filter input box. To update the list of results using that filter, press the enter key or click on the reload symbol next to the Filter input box. The list of hits is automatically updated in the Processes window.

Each of the different columns in the table can also be sorted, allowing you, for example, to list the processes in ascending or descending order depending on the process name or status or the project to which it belongs.

You can adjust the way how the processes are displayed in tabular form. You can include other columns in the table, e.g. the identifier and the date on which the process was created. You can also add selection boxes that allow the user to select individual processes for batch actions. If you choose this option, you will be able to activate a checkbox in each row of the process table. This means that you can then apply any of the actions to those processes whose checkboxes are activated rather than to the whole set of filtered processes or to the processes listed on the current page.

You can also view the workflow details for each process by clicking on the Process title. This allows you to check the current status of individual steps of the workflow. Goobi will display a small summary view of the selected process indicating the current status of each workflow step for that process. If you hold the cursor over the small square coloured symbols to the right of each workflow step, you will see a brief overview indicating which users have previously worked on that step and when.

The harvester can be used to automatically import data from external repositories.

Overview

To be able to access the harvester, the user must have the Edit harvester repositories right. The Harvester menu entry is then available under the Administration menu item. This opens the screen for listing all configured repositories.

The function Add repository opens the editing screen to create a new repository.

Configuration

The first step is to enter a name and select the protocol type. The following are available: OAI-PMH, Internet Archive Web Search, Internet Archive CLI and the BACH API.

For BACH, the URL to the BACH server and the authentication token must be specified.

If the Internet Archive Web Search is selected, the URL to the advanced search interface must be specified. To import only certain works, a search filter must also be specified as part of the URL. This way, only publications that are marked as Open Access and have been published can be imported.

In order to import access-protected publications, the Internet Archive CLI must be used. The CLI must be installed for this, usually under the path /usr/local/bin/ia. In addition, the environment variables IA_USERNAME and IA_PASSWORD must be set. A search filter can also be specified here to narrow down the hit list.

For OAI-PMH, the URL to the OAI server must be specified. If the URL contains the parameters set and format, this information is automatically determined together with the base URL. Otherwise, they must be specified manually.

With OAI, the From and Until parameter can also be set to limit the query to a specific time period. If the fields are empty, the entire period since the last request is automatically queried.

Test mode can also be activated. In this case, only the first records of the hit list are imported without the resumptionToken being analysed.

The other settings then apply to all types.

The Poll frequency defines the intervals at which the repository should be queried. The specification is in hours.

Delay defines a time period up to which new data is to be queried. If a number greater than 0 is entered here, a search will not look for all data up to the current date, but for data published up to the configured number of days before the current date.

The ‘Download folder’ field is used to specify the folder in which the data is to be downloaded and saved. The folder is created automatically during the first harvesting if it does not yet exist.

Optionally, a script can then be called that is executed on each downloaded file. This can be used, for example, to perform an XSL transformation on each XML file or to write additional information in all JSON files.

If the data is not only to be downloaded but also imported as Goobi processes, the checkbox to Create processes must be activated.

The Project, Process template and Import format to be used can then be specified.

Manual harvesting

To start harvesting manually, you can use the now run once button in the Actions column of the overview. If the project is active, harvesting is then started once.

Automatic Harvesting

Automatic harvesting takes place regularly. The time at which it should run must be defined for this. This is done in the goobi_config.properties file using the line harvesterJob=0 0 */1 * * * ?. This causes the check to take place every hour on the hour. The configuration is carried out in chron syntax and allows any time periods.

When the check is performed, it is checked for each configured active repository whether the last run was longer ago than the value configured in the poll frequency field. If this is the case, harvesting is started.

Harvesting

When a new harvest is triggered, the records that have been published or updated in the repository since the last run are determined first. For each record, Goobi checks whether it has already been processed once or is new. New files are then downloaded to the configured folder. If a script has been configured, it is called for each downloaded file.

If configured, the files are now imported. In the case of marc-xml or pica-xml, the document type is determined first. Higher-level data such as journal titles or multi-volume works are skipped. In the case of subordinate documents (journal issues, volumes of a multivolume work), the superordinate work is searched for and also downloaded. The metadata is then parsed on the basis of the ruleset from the configured process template.

The process title is created on the basis of the identifier.

This section includes several topics as for example configuration and granular adaption for data protection as well as information about metadata and mappings.

As a web-based application, Goobi has its own structure and is located on a defined path in the file system independently of the servlet container being used. This section explains how to organise the directory structures within which Goobi saves its data and the different configuration files.

The base path for all digitisation software in the Goobi environment is:

/opt/digiverso/

The following directories are usually located on this base path:

/opt/digiverso/goobi/
/opt/digiverso/logs/
/opt/digiverso/itm/
/opt/digiverso/viewer/

The logs directory is the main directory for log files. Goobi log files are also stored here (assuming the system is properly configured). The other directories listed above relate to frequently used applications (e.g. viewer for the Goobi viewer, itm for the intranda Task Manager and goobi for Goobi.

The base path for Goobi is:

/opt/digiverso/goobi/

In most cases, this base path will accommodate the following folder structure (see below for details of each sub-directory):

/opt/digiverso/goobi/config/
/opt/digiverso/goobi/import/
/opt/digiverso/goobi/metadata/
/opt/digiverso/goobi/plugins/
/opt/digiverso/goobi/rulesets/
/opt/digiverso/goobi/scripts/
/opt/digiverso/goobi/xslt/

Depending on the way Goobi has been installed, the import directory will contain a range of data, mostly on a temporary basis. By way of example, import plug-ins use this directory to enter metadata and associated digital content in order to create processes. The respective import plug-ins are also responsible for deleting files that are no longer needed.

The config directory contains all the Goobi configuration files that do not have to be located within the application itself. These are listed below:

config_contentServer.xml
goobi_activemq.xml
goobi_config.properties
goobi_digitalCollections.xml
goobi_exportXml.xml
goobi_mail.xml
goobi_metadataDisplayRules.xml
goobi_normdata.xml
goobi_opac.xml
goobi_opacUmlaut.txt
goobi_processProperties.xml
goobi_projects.xml
goobi_rest.xml
goobi_webapi.xml
messages_de.properties
messages_en.properties

Depending on the specific installation, the config directory may also contain other configuration files in addition to those related to the application’s core components. Accordingly, we recommend that you also use this central configuration directory to store configurations for individual plug-ins that provide additional functionality.

plugin_abc.xml
plugin_xyz.xml

For subsequent ease of maintenance, the paths and file names relating to the configuration of any new Goobi plug-ins that may be developed should also adhere to this convention.

The metadata directory is the central directory for storing Goobi metadata and digital content. Within this directory there is a directory with the name of the Goobi process ID for each Goobi process. The directories for the individual Goobi processes are organised as follows:

└── 1234
    ├── export
    │   └── ...
    ├── images
    │   └── ...
    ├── import
    │   └── ...
    ├── ocr
    │   └── ...
    ├── taskmanager
    │   └── ...
    ├── thumbs
    │   └── ...
    ├── validation
    │   └── ...
    └── meta.xml

In addition to the central metadata file meta.xml, there may be other backup files such as meta.xml-2021-11-28-175618166, meta.xml.2024-05-08-135432256, etc. in the directory, depending on the configuration.

Directory images

The images directory is temporarily made available to various users within the workflow and has the following structure:

└── 1234
    ├── images
    │   └── abc_8765432_raw
    │   │   └── 00000001.raw
    │   │   └── 00000002.raw
    │   │   └── 00000003.raw
    │   │   └── 00000004.raw
    │   │   └── 00000005.raw
    │   ├── abc_8765432_master
    │   │   └── 00000001.tif
    │   │   └── 00000002.tif
    │   │   └── 00000003.tif
    │   │   └── 00000004.tif
    │   │   └── 00000005.tif
    │   └── abc_8765432_media
    │   │   └── 00000001.jpg
    │   │   └── 00000002.jpg
    │   │   └── 00000003.jpg
    │   │   └── 00000004.jpg
    │   │   └── 00000005.jpg
    │   └── abc_8765432_source
    │       └── source_files_1.pdf
    │       └── source_files_2.xls
    │       └── source_files_3.wmv
    └── meta.xml

Subdirectory media

The most important folder for working with the content is the one that ends in _media. Files that are to be used for publication (e.g. in a Goobi viewer) are stored here. The files stored here are usually compressed derivatives that can be used for working with the digitised material and publishing it in very good quality.

Subdirectory master

The master files are located in the directory that ends with _master. These are usually the unaltered originals, such as those generated by scanners. They are typically uncompressed and not optimised in any other way (straightened, cropped or similar).

Subdirectory source

In the directory whose name ends in _source, source files can be stored to which Goobi users should also have access in order to upload or view them. The files stored here are taken into account and exported as part of the standard Goobi workflow export. In the case of export to the Goobi viewer, for example, this means that the data from the source folder is also exported to the Goobi viewer's hotfolder.

Further subdirectories

OCR

In addition to the images directory, there may also be an ocr directory. This contains all the OCR results that were generated within the workflow and added to the process. There is a separate directory with the respective files in it for each existing format of the OCR results.

└── 1234
    └── ocr
        └── abc_8765432_alto
        │   └── 00000001.xml
        │   └── 00000002.xml
        │   └── 00000003.xml
        │   └── 00000004.xml
        │   └── 00000005.xml
        ├── abc_8765432_doc
        │   └── 00000001.doc
        │   └── 00000002.doc
        │   └── 00000003.doc
        │   └── 00000004.doc
        │   └── 00000005.doc
        └── abc_8765432_pdf
            └── 00000001.pdf
            └── 00000002.pdf
            └── 00000003.pdf
            └── 00000004.pdf
            └── 00000005.pdf

Thumbnails

Smaller versions of the images in images can be saved in the thumbs folder, which Goobi uses to display the images in low resolution. This significantly increases the speed of image display for larger images. For each subfolder of images, one or more subfolders can be created in thumbs with the same name as the images subfolder, extended by an additional underscore _ and a size specification in pixels. This size specification must correspond to the maximum height and width of the images in the respective subfolder. The file names of the images in the thumbs subfolder must correspond to those of the images in the corresponding images subfolder, but with the file extension .jpg.

└── 1234
    └── thumbs
        └── abc_8765432_master_800
        │   └── 00000001.jpg
        │   └── 00000002.jpg
        │   └── 00000003.jpg
        │   └── 00000004.jpg
        │   └── 00000005.jpg
        ├── abc_8765432_master_3000
        │   └── 00000001.jpg
        │   └── 00000002.jpg
        │   └── 00000003.jpg
        │   └── 00000004.jpg
        │   └── 00000005.jpg
        ├── abc_8765432_media_800
        │   └── 00000001.jpg
        │   └── 00000002.jpg
        │   └── 00000003.jpg
        │   └── 00000004.jpg
        │   └── 00000005.jpg
        └── abc_8765432_media_3000
            └── 00000001.jpg
            └── 00000002.jpg
            └── 00000003.jpg
            └── 00000004.jpg
            └── 00000005.jpg

If there are matching images in thumbs for an image file in images, then these are automatically used in Goobi to display thumbnails and zoomable images when zoomed out.

Validation

If automatic validation, e.g. of images, takes place on the Goobi server and also in the workflows used, the validation directory exists.

Within this directory, a subfolder is created for each validation run so that older validation results can also be stored. The generated subfolders are always named in such a way that the folder name lists the date and time as well as the type of validation. This looks as follows, for example:

└── 1234
    └── validation
        └── 2022-11-20_11-20-01_jpylyzer
        │   └── 00000001.xml
        │   └── 00000002.xml
        │   └── 00000003.xml
        │   └── 00000004.xml
        │   └── 00000005.xml
        ├── 2022-11-20_12-02-13_jhove
        │   └── 00000001.xml
        │   └── 00000002.xml
        │   └── 00000003.xml
        │   └── 00000004.xml
        │   └── 00000005.xml
        └── 2022-11-23_08-12-56_jpylyzer
            └── 00000001.xml
            └── 00000002.xml
            └── 00000003.xml
            └── 00000004.xml
            └── 00000005.xml

If there is collaboration with the intranda TaskManager, the ‘taskmanager’ directory also exists within the folder. Within this directory, the TaskManager saves temporary data for the execution of long-running tasks. Depending on the configuration, all ticket and template files used for the individual TaskManager calls are also permanently saved and stored here. The content of this directory is as follows:

└── 1234
    └── taskmanager
        └── 2022-11-23_08-12-56_jp2validate
        │   └── 00000001.xml
        │   └── 00000002.xml
        │   └── 00000003.xml
        │   └── 00000004.xml
        │   └── 00000005.xml
        └── 2022-11-25_14-38-15_iii-create_jpeg
            └── 00000001.jpeg
            └── 00000002.jpeg
            └── 00000003.jpeg
            └── 00000004.jpeg
            └── 00000005.jpeg

Import

Depending on the individual installation, there is also an import folder for each Goobi process. This folder is used by import plug-ins to carry original source files associated with the respective process. Imported and imported catalogue data record files or other source files can be stored here and used within scripts as part of workflow processing. The contents of the folder could look like this, for example:

└── 1234
    └── import
        └── eod
        │   └── 00000001.tif
        │   └── 00000002.tif
        │   └── 00000003.tif
        │   └── 00000004.tif
        │   └── 00000005.tif
        ├── abc.mrc
        └── abc.original.pdf

Export

A folder with the name ‘export’ can also exist within the process directory. Files with the same structure as those in the images directory can be saved in this folder. The files in this folder are then included in the export and exported to the Goobi viewer hotfolder, for example. This may be necessary if files are to be included in the export without them being present in the images folder and without them being referenced within the METS file.

The content could look like this, for example:

└── 1234
    └── export
        └── abc_8765432_media
            └── intro.mp3
            └── overview.jpg
            └── some_file.pdf

Goobi can also use variables when performing certain actions. Some of these variables can be used only in specific contexts, e.g. the variables {login} and {user full name} when configuring the LDAP.

There are also ‘static’ and ‘dynamic’ variables. These can be used at various points, e.g. when calling scripts or configuring projects.

Static variables

Static variables are variables with a fixed name and a defined value.

Static variable: {prefs}

Static variable: {processid}

Static variable: {processtitle}

Static variable: {stepid}

Static variable: {stepname}

Static variable: {processpath}

Static variable: {importpath}

Static variable: {imagepath}

Static variable: {tifpath}