Importing records from an Excel file

Introduction
This documentation describes the installation, configuration and use of the plugin for mass importing data sets from Excel files.
Details
Text
Identifier
intranda_import_excel
Source code
​https://github.com/intranda/goobi-plugin-import-excel​
Licence
GPL 2.0 or newer
Compatibility
Goobi workflow 2020.09
Documentation date
11.11.2020
Installation
The plugin must be installed in the following folder:
1
/opt/digiverso/goobi/plugins/import/plugin_intranda_import_excel.jar
Copied!
There is also a configuration file, which must be located at the following place:
1
/opt/digiverso/goobi/config/plugin_intranda_import_excel.xml
Copied!
Configuration
The configuration is done via the file plugin_intranda_import_excel.xml. This file can be adapted during operation.
1
<config_plugin>
2
    <config>
3
        <!-- which workflow template shall be used -->
4
        <template>*</template>
5
​
6
        <!-- publication type to create -->
7
        <publicationType>Monograph</publicationType>
8
​
9
        <!-- which digital collection to use -->
10
        <collection>mycollection</collection>
11
​
12
        <!-- define if a catalogue shall get requested to import metadata -->
13
        <useOpac>true</useOpac>
14
        <!-- which catalogue to use (as default) -->
15
        <opacName>GBV PICA</opacName>
16
        <!-- which catalogue to use per record; if missing the default will be used -->
17
        <opacHeader>Catalogue</opacHeader>
18
        <searchField>12</searchField>
19
​
20
        <!-- define in which row the header is written, usually 1 -->
21
        <rowHeader>1</rowHeader>
22
        <!-- define in which row the data starts, usually 2 -->
23
        <rowDataStart>2</rowDataStart>
24
        <!-- define in which row the data ends, usually 20000 -->
25
        <rowDataEnd>20000</rowDataEnd>
26
​
27
        <!-- define which column is the one to use for catalogue requests -->
28
        <identifierHeaderName>PPN-A</identifierHeaderName>
29
​
30
        <!-- Rules to generate the process title, the same syntax as in goobi_projects.xml can be used.
31
            Use the column names to get the right metadata values.
32
            If the field is missing or empty, the value of CatalogIDDigital is used. -->
33
        <processTitleRule>2-Titel+'_'+PPN-O</processTitleRule>
34
​
35
        <!-- prefix path to the image folder. Can be empty or missing if the import doesn't contain images or if the excel field contains absolute path  -->
36
        <imageFolderPath>/opt/digiverso/images/</imageFolderPath>
37
        <!-- define which column contains the image folder name. Can be combined with <imageFolderPath> prefix or an absolute path.
38
        If the field is missing, empty or does not contain an existing directory, no images will be imported -->
39
        <imageFolderHeaderName>images</imageFolderHeaderName>
40
​
41
        <!-- defines, if images are moved from the source folder to the destination (true) or copied (false) -->
42
        <moveFiles>true</moveFiles>
43
​
44
        <!-- Run the import as GoobiScript -->
45
        <runAsGoobiScript>true</runAsGoobiScript>
46
​
47
        <!-- Overwrite any existing processes -->
48
        <replaceExistingProcesses>true</replaceExistingProcesses>
49
​
50
        <!-- define here which columns shall be mapped to which ugh metadata
51
            ugh: name of the metadata to use. if it is empty or missing, no metadata is generated
52
            headerName: title inside of the header column
53
            property: name of the process property. if it is empty or missing, no process property gets generated
54
            normdataHeaderName: title of the header column to use for a gnd authority identifier
55
            docType: define if the metadata should be added to the anchor or child element. Gets ignored, when the
56
            record is no multivolume. Default is 'child', valid values are 'child' and 'anchor' -->
57
        <metadata ugh="CatalogIDSource" headerName="PPN-A" />
58
        <metadata ugh="CatalogIDDigital" headerName="PPN-O" />
59
        <metadata ugh="TitleDocMain" headerName="2-Titel" />
60
        <metadata ugh="PlaceOfPublication" property="Ort" normdataHeaderName="4-GND-ORT" headerName="3-Ort" docType="anchor" />
61
        <metadata ugh="DocLanguage" headerName="10-DocLanguage" />
62
​
63
        <!-- a configuration for a person might look like this -->
64
        <person ugh="Author" normdataHeaderName="7-GND-Person" docType="child">
65
            <!-- use this field if the column contains the complete name -->
66
            <nameFieldHeader>11-Person</nameFieldHeader>
67
            <!-- set this field to true, if the name must be splitted into first- and lastname. The complete name gets written into lastname -->
68
            <splitName>true</splitName>
69
            <!-- define at which character the name is separated. @firstNameIsFirstPart defines, if the firstname is the first or last part of the name -->
70
            <splitChar firstNameIsFirstPart="false">, </splitChar>
71
​
72
            <!-- use this fields, if the firstname and lastname are in different columns -->
73
            <!--
74
            <firstname>5-Vorname</firstname>
75
            <lastname>6-Nachname</lastname>
76
            -->
77
        </person>
78
​
79
    </config>
80
​
81
    <config>
82
        <!-- which workflow template shall be used -->
83
        <template>json_opac_import</template>
84
​
85
        <!-- publication type to create -->
86
        <publicationType>Monograph</publicationType>
87
        <collection>DefaultCollection</collection>
88
​
89
        <useOpac>true</useOpac>
90
        <opacName>ArchiveSpace</opacName>
91
        <opacHeader>Catalogue Name</opacHeader>
92
​
93
        <!-- define in which row the header is written, usually 1 -->
94
        <rowHeader>1</rowHeader>
95
        <!-- define in which row the data starts, usually 2 -->
96
        <rowDataStart>2</rowDataStart>
97
        <!-- define in which row the data ends, usually 20000 -->
98
        <rowDataEnd>20000</rowDataEnd>
99
​
100
        <processTitleRule>aspace_uri+bib_id+'_'+barcode+holdings+item</processTitleRule>
101
​
102
        <!-- Run the import as GoobiScript -->
103
        <runAsGoobiScript>false</runAsGoobiScript>
104
​
105
        <metadata opacSearchField="ao" headerName="aspace_uri" />
106
        <metadata opacSearchField="bib" headerName="bib_id" />
107
        <metadata opacSearchField="type" headerName="barcode" />
108
        <metadata opacSearchField="type" headerName="holdings" />
109
        <metadata opacSearchField="type" headerName="item" />
110
    </config>
111
</config_plugin>
Copied!
Individual configurability
It is possible to create a global configuration for all production templates as well as individual settings for individual production templates. The element config can be repeated in the XML file. If mass import has been selected in Goobi, the system always searches for the configuration block with the name of the selected production template in the template element. If such an entry does not exist, the default configuration is used. This is marked with *.
1
<!-- which workflow template shall be used -->
2
<template>*</template>
Copied!
Collection
With the optional element collection it is possible to define a collection to be inserted into all records. In addition, collections can also be selected from the interface, or the collection can be imported as part of the Excel file or from the catalog.
1
<!-- which digital collection to use -->
2
<collection>Example collection</collection>
Copied!
Catalogue import
The next four elements use-Opac, opacName, opacHeader and searchField control whether a catalogue query should be performed during the import. If useOpac contains the value true, such a query is performed. The catalogue and the search field configured in the fields are used for this. The name of the catalogue must correspond to an entry in the Goobi configuration file goobi_projects.xml. It can either be permanently defined in the opacName parameter or used dynamically from a line of the relevant record (the opacHeader). The structure type is automatically recognised by the OPAC data.
1
<!-- define if an opac request is made -->
2
<useOpac>true</useOpac>
3
<!-- name of the configured catalogue -->
4
<opacName>K10Plus</opacName>
5
<!-- which catalogue to use per record; if missing the default will be used -->
6
<opacHeader>Catalogue</opacHeader>
7
<!-- field to search in -->
8
<searchField>12</searchField>
Copied!
However, if no OPAC is used, the structure type of the operations to be created must be specified in the publicationType field. The name used here must exist within the ruleset. If the OPAC is to be used, this field is not evaluated.
1
<!-- publication type to create -->
2
<publicationType>Monograph</publicationType>
Copied!
Line range
The following elements describe the structure of the Excel file to be imported.
rowHeader defines the row in which the column headings that are later relevant for the mapping were entered. This is usually the first line. However, this can also differ for multi-line entries.
1
<!-- define in which row the header is written, usually 1 -->
2
<rowHeader>1</rowHeader>
Copied!
rowDataStart and rowDataEnd describe the area that contains the data. Usually, these are the lines that follow the rowHeader directly, but special formatting can also contain blank lines that can be removed using this.
1
<!-- define in which row the data starts, usually 2 -->
2
<rowDataStart>2</rowDataStart>
3
<!-- define in which row the data ends, usually 20000 -->
4
<rowDataEnd>20000</rowDataEnd>
Copied!
Identifier
The identifierHeaderName entry contains the heading of the column in which an identifier is contained. This field is used internally to identify the rows. This value is used for an OPAC query. In addition, this value is also used to generate the transaction title if no other generation has been specified for the transaction title.
1
<!-- define which column is the one to use for catalogue requests and to identify the row during the import -->
2
<identifierHeaderName>Identifier</identifierHeaderName>
Copied!
Process title
The element processTitleRule is used to generate the operation title. The same options are available here that can also be used in the Goobi configuration file goobi_projects.xml.
1
<!-- Rules to generate the process title, the same syntax as in goobi_projects.xml can be used.
2
     Use the column names to get the right metadata values.
3
     If the field is missing or empty, the value of the identifier column is used.
4
-->
5
<processTitleRule>'StaticPrefix_'+Identifier</processTitleRule>
Copied!
Transfer of images
The elements imageFolderHeaderName, imageFolderPath and moveFiles can be used to import images in addition to metadata. In imageFolderHeaderName the column name is entered, in which the folder names containing the images can be found in the Excel file. Either an absolute path or a relative path can be specified there.
If a relative path is specified, the element imageFolderPath must contain the root path to the images. The element moveFiles can be used to control whether the images should be copied or moved.
1
<!-- define which column contains the image folder name. Can be combined with <imageFolderPath> prefix or an absolute path.
2
      If the field is missing, empty or does not contain an existing directory, no images will be imported -->
3
<imageFolderHeaderName>image folder</imageFolderHeaderName>
4
​
5
<!-- prefix path to the image folder. Can be empty or missing if the import doesn't contain images or if the excel field contains absolute path  -->
6
<imageFolderPath>/mnt/images/</imageFolderPath>
7
​
8
<!-- defines, if images are moved from the source folder to the destination (true) or copied (false) -->
9
<moveFiles>true</moveFiles>
Copied!
Execution using GoobiScript
The element runAsGoobiScript controls whether an import should be processed asynchronously in the background via the GoobiScript queue or whether the import should be processed directly within the user session. Here you have to consider which setting makes sense. If an import including images is to take place or if the Excel file contains a lot of data records, it probably makes more sense to perform this import as GoobiScript.
1
<!-- Run the import as GoobiScript -->
2
<runAsGoobiScript>true</runAsGoobiScript>
Copied!
Note: If the identifierHeaderName column does not contain a unique identifier or has not been configured, the runAsGoobiScript option cannot be used.
Configuration of the individual Excel columns
The fields metadata, person and group can be used to import individual columns as metadata or as process properties. Each field contains a number of attributes and sub-elements.
Import metadata
The metadata element is used to generate descriptive metadata..
Name
Type
Description
headerName
Attribut
Column header in the Excel file
ugh
Attribut
Name of the metadata
property
Attribut
Name of the property
docType
Attribut
anchor or child
normdataHeaderName
Attribut
Column header of a column with associated identifiers
opacSearchField
Attribut
Definition of which search field should be used for the catalogue query. This is necessary for the use of the JSON-Opac-Plugin.
The headerName attribute contains the column header. The rule only applies if the Excel file contains a column with this title and the cell is not empty. At least one of the two attributes ugh and name must exist. The ugh field can contain the name of a metadata. If this is the case (and the metadata is allowed for the configured publication type), a new metadata is created. name creates a property with this name.
The docType attribute becomes relevant if a multi-volume work or journal has been imported from the catalog. This can be used to control whether the field should belong to the entire recording or to the volume.
If, in addition to the content, there is another column with standard data identifiers or URIs, this column can be added to the normdataHeaderName attribute.
Import of persons
The person element can be used to automatically create persons.
Name
Type
Description
ugh
Attribut
Name of the person role
docType
Attribut
anchor or child
normdataHeaderName
Attribut
Column header of a column with associated identifiers
firstnameFieldHeader
Element
Column header of field for first name
lastnameFieldHeader
Element
Column header for surnames
nameFieldHeader
Element
Column header for the complete name
splitName
Element
Defines whether the value in nameFieldHeader should be split.
splitChar
Element
Element at which splitting takes place. Default is the first space character.
firstNameIsFirstPart
Attribut
Defines the order in which the data was entered.
Persons differ from normal metadata in that they consist of first and last names. This specification can be in two different columns, then the elements firstnameFieldHeader and lastnameFieldHeader are used. If the names are only in one column, the field nameFieldHeader is used. In this case, the system checks whether the specifications should only contain the surname or whether the content must be split. With splitChar you can set the character/sequence at which the splitting should take place. The attribute firstNameIsFirstPart contains the information whether the name is to be imported as First name Last name or Last name First name.
Import of metadata groups
Metadata groups can be created using the group element.
Name
Type
Description
ugh
Attribut
Name of the metadata group
docType
Attribut
anchor or child
metadata
Element
Metadata within the group
person
Element
Person within the group
A metadata group consists of several metadata and persons. The configuration of the individual sub-elements is identical to that of the individual metadata and persons.
Usage in Goobi
To use the import, the mass import area must be opened in the production templates and the plugin intranda_import_excel selected in the File upload import tab. An Excel file can then be uploaded and imported.
The import then takes place line by line. A new process is created for each line and the configured rules are applied. If a valid data record has been created and the generated task title has not yet been assigned, the task is actually created and saved.