Goobi workflow Plugins
Documentation homeGoobi workflow ManualGoobi workflow Digests
English
English
  • Overview
  • Administration
    • Archive Management
      • Using the plugin
      • Configuration of the plugin
    • Configuration editor
    • Copy Master-Anchor
    • Data Poller
    • Goobi-to-Goobi
      • Installation and configuration
      • Creation of the export directories
      • Transfer of the export directories
      • Importing the export directories
    • Reset pagination
    • Restoring archived image folders
    • Ruleset Compatibility
    • Ruleset editor
  • Dashboards
    • Barcode scanner Dashboard
    • Extended Dashboard
  • Exports
    • Customised export for the DMS Imagen Media Archive Management
    • Single Page Export
    • Configurable export
    • Fedora Export
    • Fedora Export PROV
    • Heris Export
    • Export for newspapers to the portal of the German Digital Library
    • PDF export to the NLI directory structure
    • Export of selected images
    • Stanford Export
    • VLM Export
    • HAAB Export
    • ZOP Export
  • Generic
    • Barcode Scanner
  • Imports
    • Legacy data import for the Austrian Federal Monuments Authority
    • Archive data import
    • Data import without catalogue query for ETH Zurich
    • Importing records from an Excel file
    • Import of card catalogues from KatZoom
    • Importing MAB Files
    • Import of Sisis SunRise Files
    • Import for journal articles from an Endnote Export
    • Data import with ALMA catalogue query for Zurich Central Library
    • Data import with CMI catalogue query for Zurich Central Library
    • Data import without catalogue query for the Zentralbibliothek Zurich
  • Metadata
    • Change Publication Type
    • Metadata extension for the creation of structural elements per image
  • OPAC
    • Ariadne Import
    • EAD data transfer
    • Generic XML Import
    • Generic JSON Import
    • Kalliope Import
    • MARC Import
    • PICA Import
    • Soutron Import
  • Repeated Jobs
    • Data import for the Austrian Housing Promotion Fund
    • HERIS Vocabulary Update
  • Statistics
    • Sudan Memory Translations
    • Visualisation of the throughput per user
  • Steps
    • ALMA API Plugin
    • Automatic pagination based on file names
    • Archiving image folders
    • Generating Archival Resource Keys (ARK)
    • Libsafe Integration
    • Assign batch
    • Batch Progress Plugin
    • Catalogue Request
    • Changing the workflow based on process properties
    • Generation of PDF files
    • Plugin for registering DOI via the DataCite API
    • Delay Workflow
    • Conditional workflow status delay
    • Delete Content
    • Display of metadata in a task
    • Plugin for DOI registration
    • Downloading and verifying files
    • Duplication of work steps
    • ePIC PID Registration (Handle & DOI)
    • EWIG Long term archiving
    • Metadata enrichment via Excel file
    • Package Export
    • Copying files from metadata fields
    • Upload files
    • File validation
    • Flex Editor
    • Generate ALTO IDs
    • Generate Identifier
    • Geonames Annotation
    • GeoNames Correction
    • Automatic Handle Assignment
    • Heris data import
    • Extraction of image metadata
    • Image scaling and watermarking
    • Selection of images
    • Quality control of images
    • Metadata transfer from a directory
    • Metadata Cleaning
    • Layout Wizzard
      • Using the plugin
        • Preview
          • Image area
          • Display and navigation options
        • Single page view
          • Folder and file options
          • Current image
          • General settings
          • File list
          • Save view
          • Working steps
          • Selected analysis step
          • Global cutting options
      • Technical details
        • Installation
        • Configuration of the LayoutWizzard
        • Configuration of the user interface
        • Workflow
    • Metadata edition
    • Capture metadata per image
    • Automatic enrichment of metadata from own vocabularies
    • Structure data import from an Excel file
    • Update Metadata Fields
    • Automatic METS enrichment with image files and pagination
    • Enrich METS file
    • Data migration from Visual Library
    • MIX Metadata Enrichment Plugin
    • OCR execution with mixed fonts
    • OCR page selection
    • Transfer OCR result to metadata field
    • Object Identifier Generation
    • Correction of tables of contents after an OLR
    • Data import for Book Interchange files
    • Split PDFs, extract full text and read table of contents
    • Electronic Publications
    • Generation of placeholder images
    • Process folder migration
    • Renaming files
    • Renaming files before the Rosetta ingest
    • Renaming Processes
    • Reorder Images
    • Replace images
    • Automatic setting of the representative
    • Reverse Image Order
    • Generation of docket files
    • Sending emails
    • Import of ECHO files as TEI
    • Tif-Validation
    • Transcription of image content
    • OCR using Transkribus
    • Import and download from Transkribus Collections
    • Creation of Uniform Resource Names (URN)
    • User Assignment
    • Vocabulary enrichment
    • Writing XMP metadata to image files
    • Metadata validation within a task
    • Invoices and delivery notes for user orders
  • Workflow
    • AEON data transfer
    • Barcode Generator
    • Close steps
    • Entity Editor - Artist Dictionary
    • Generic import plugin for excel files including validation
    • Process creation through file upload
    • Mass import from Excel data with EAD enrichment
    • Generic import plugin for JSON files
    • LayoutWizzard workflow plugin
    • Create process relationships
    • Mass upload
    • Import of newspaper issues as single pages
    • Project export as folder with images and Excel file
    • Mass import for brand studies and advertising material
    • Data transfer from AIM25
Powered by GitBook
On this page
  • Overview
  • Introduction
  • Installation
  • Overview and functionality
  • Configuration
Export as PDF
  1. Workflow

Import of newspaper issues as single pages

This workflow plugin allows a mass import of newspaper issues in the form of individual pages, in which the date, issue numbers and page numbers are transferred.

PreviousMass uploadNextProject export as folder with images and Excel file

Last updated 4 months ago

Overview

Name
Wert

Identifier

intranda_workflow_newspaper_pages_importer

Repository

Licence

GPL 2.0 or newer

Last change

13.07.2024 09:56:26

Introduction

This workflow plugin allows the mass import of newspaper issues that are available as individual pages. For each file in a folder, the issue date and issue number are determined based on the file name. Goobi processes are then created at year level and the issues are generated together with metadata and page associations.

Installation

To install the plugin, the following two files must be installed:

/opt/digiverso/goobi/plugins/workflow/plugin-workflow-newspaper-pages-importer-base.jar
/opt/digiverso/goobi/plugins/GUI/plugin-workflow-newspaper-pages-importer-gui.jar

To configure how the plugin should behave, various values can be adjusted in the configuration file. The configuration file is usually located here:

/opt/digiverso/goobi/config/plugin_intranda_workflow_newspaper_pages_importer.xml

To use this plugin, the user must have the correct role authorisation.

Therefore, please assign the role Plugin_workflow_newspaper_pages_importer to the group.

Overview and functionality

If the plugin has been installed and configured correctly, it can be found under the 'Workflow' menu item.

After entering the plugin, the actual import process can be started. This involves searching for existing files within the configured source directory and checking their names. The naming scheme within the file name must look like this:

yyyy-MM-dd_AAA.bbb

The characters are in favour of the following:

Character
Explanation

yyyy

Specification of the four-digit year

MM

Specification of the two-digit month, with leading zero if necessary

dd

Specification of the two-digit day, with leading zero if necessary

AAA

Numerical issue number in three digits, with leading zeros if necessary

bbb

File extension, e.g. pdf, jpeg or tif

An example of a directory listing for such a folder content:

tree /opt/digiverso/import
/opt/digiverso/import
├── 1867-04-06_001.pdf
├── 1867-04-06_002.pdf
├── 1867-04-06_003.pdf
├── 1867-04-06_004.pdf
├── 1867-04-20_001.pdf
├── 1867-04-20_002.pdf
├── 1867-04-20_003.pdf
├── 1867-04-20_004.pdf
├── 1867-05-04_001.pdf
├── 1867-05-04_002.pdf
├── 1867-05-04_003.pdf
├── 1867-05-04_004.pdf
├── 1867-05-11_001.pdf
├── 1867-05-11_002.pdf
├── 1867-05-11_003.pdf
├── 1867-05-11_004.pdf
├── 1867-05-18_001.pdf
├── 1867-05-18_002.pdf
├── 1867-05-18_003.pdf
├── 1867-05-18_004.pdf
├── 1867-05-25_001.pdf
├── 1867-05-25_002.pdf
├── 1867-05-25_003.pdf
├── 1867-05-25_004.pdf

During the import process, a process is created in Goobi for each year, in which a structural element is created for each newspaper issue with the associated data, which is generated from the file names and the values of the configuration.

Configuration

The plugin is configured in the file plugin_intranda_workflow_newspaper_pages_importer.xml as shown here:

<config_plugin>
	

	<set title="Block 1">

		<!-- which folder to use as import source -->
		<importFolder>/opt/digiverso/import/sample1/</importFolder>
		
		<!-- which workflow to use -->
		<workflow>Newspaper_workflow</workflow>
		
		<!-- prefix for the process title; will be extended by '_' and the year information -->
		<processtitle>mytitle_1234567</processtitle>
		
		<!-- prefix to use for the issue titles -->
		<issueTitlePrefix>Ausgabe vom</issueTitlePrefix>		
		<issueTitlePrefixMorning identifier="_a_">Morgenausgabe vom</issueTitlePrefixMorning>		
		<issueTitlePrefixEvening identifier="_b_">Abendausgabe vom</issueTitlePrefixEvening>		
	
		<!-- prefix for the page labels -->
		<pageNumberPrefix>Page </pageNumberPrefix>
	
		<!-- language to use for long date formatter in issue title -->
		<languageForDateFormat>de</languageForDateFormat>
	
	
		<!-- Whether or not to delete the images from the import folder once they are imported. OPTIONAL. DEFAULT false. -->
		<deleteFromSource>true</deleteFromSource>

		<!-- Configure here the metadata that shall be added to the anchor file or the volume part of the mets file. -->
		<!-- This tag accepts the following attributes:
			- @value: metadata value template, which may contain a variable defined by @var wrapped with _ from both sides
			- @type: metadata type
			- @var: variable that is ready to be used in @value. To use it, wrap it with _ from both sides and put it into the @value string. OPTIONAL.
						- Options are YEAR | MONTH | DAY | DATE | DATEFINE, where cases only matters for the references in @value string.
						- The only difference between DATE and DATEFINE are their representations of the date: DATE keeps the original format "yyyy-mm-dd" while DATEFINE takes a new one "dd. MMM. yyyy".
						- The value of an unknown variable will be its name.
						- For example, 
							- IF one defines a @var by "year", then it should be referenced in @value using "_year_"
							- IF one defines a @var by "YEAR", then it should be referenced in @value using "_YEAR_", although YEAR and year are actually the same option
							- IF one defines a @var by "unknown", then any occurrences of "_unknown_" will be replaced by "unknown"
							- IF one wants to have a @value template containing "_Year_" as hard-coded, then "Year" should be avoided to be @var. If in such cases such a 
							  variable is still needed, then one can define @var to be something like "yEaR".
			- @anchor: true if this metadata is an anchor metadata, false if not. OPTIONAL. DEFAULT false.
			- @volume: true if this metadata is a volume metadata, false if not. OPTIONAL. DEFAULT false.
			- @person: true if this metadata is a person metadata, false if not. OPTIONAL. DEFAULT false.
		 -->
		<metadata value="CHANGE_ME" type="TitleDocMain" anchor="true" volume="false" person="false" />
		<!-- The @volume attribute is by default false. -->
		<metadata value="CHANGE_ME" type="CatalogIDSource" anchor="true" person="false" />
		<!-- The @person attribute is by default false. -->
		<metadata value="CHANGE_ME" type="CatalogIDDigital" anchor="true" />
		
		<metadata value="CHANGE_ME" type="CurrentNoSorting" anchor="false" volume="true" />
		<!-- The @anchor attribute is by default false. -->
		<metadata value="zeitungen#livb" type="SubjectTopic" volume="true" />
		<metadata value="pt_zeitung" type="Publikationstyp" volume="true" />
		<metadata value="zeitungsherausgeber#livb" type="Classification" volume="true" />
		<metadata value="eli" type="ViewerInstance" volume="true" />
		
		<!-- Use @var to generate the final metadata values in the run. -->
		<metadata var="YEAR" value="Newspaper Volume _YEAR_" type="CatalogIDDigital" volume="true" />
		<metadata var="YEAR" value="Liechtensteiner Volksblatt (_YEAR_)" type="TitleDocMain" volume="true" />
		<metadata var="YEAR" value="_YEAR_" type="CurrentNo" volume="true" />
		<metadata var="YEAR" value="_YEAR_" type="PublicationYear" volume="true" />
	</set>
	
	<set title="Block 2">

		<!-- which folder to use as import source -->
		<importFolder>/opt/digiverso/import/sample2/</importFolder>
		
		<!-- which workflow to use -->
		<workflow>Newspaper_workflow</workflow>
		
		<!-- prefix for the process title; will be extended by '_' and the year information -->
		<processtitle>mytitle_1234567</processtitle>
	
		<!-- prefix for the page labels -->
		<pageNumberPrefix>Page </pageNumberPrefix>
	
		<!-- language to use for long date formatter in issue title -->
		<languageForDateFormat>de</languageForDateFormat>
	
		<!-- prefix to use for the issue titles -->
		<issueTitlePrefix>Ausgabe vom</issueTitlePrefix>
		<issueTitlePrefixMorning identifier="_a_">Morgenausgabe vom</issueTitlePrefixMorning>		
		<issueTitlePrefixEvening identifier="_b_">Abendausgabe vom</issueTitlePrefixEvening>		
	
		<!-- Whether or not to delete the images from the import folder once they are imported. OPTIONAL. DEFAULT false. -->
		<deleteFromSource>true</deleteFromSource>

		<!-- Configure here the metadata that shall be added to the anchor file or the volume part of the mets file. -->
		<!-- This tag accepts the following attributes:
			- @value: metadata value template, which may contain a variable defined by @var wrapped with _ from both sides
			- @type: metadata type
			- @var: variable that is ready to be used in @value. To use it, wrap it with _ from both sides and put it into the @value string. OPTIONAL.
						- Options are YEAR | MONTH | DAY | DATE | DATEFINE, where cases only matters for the references in @value string.
						- The only difference between DATE and DATEFINE are their representations of the date: DATE keeps the original format "yyyy-mm-dd" while DATEFINE takes a new one "dd. MMM. yyyy".
						- The value of an unknown variable will be its name.
						- For example, 
							- IF one defines a @var by "year", then it should be referenced in @value using "_year_"
							- IF one defines a @var by "YEAR", then it should be referenced in @value using "_YEAR_", although YEAR and year are actually the same option
							- IF one defines a @var by "unknown", then any occurrences of "_unknown_" will be replaced by "unknown"
							- IF one wants to have a @value template containing "_Year_" as hard-coded, then "Year" should be avoided to be @var. If in such cases such a 
							  variable is still needed, then one can define @var to be something like "yEaR".
			- @anchor: true if this metadata is an anchor metadata, false if not. OPTIONAL. DEFAULT false.
			- @volume: true if this metadata is a volume metadata, false if not. OPTIONAL. DEFAULT false.
			- @person: true if this metadata is a person metadata, false if not. OPTIONAL. DEFAULT false.
		 -->
		<metadata value="CHANGE_ME" type="TitleDocMain" anchor="true" volume="false" person="false" />
		<!-- The @volume attribute is by default false. -->
		<metadata value="CHANGE_ME" type="CatalogIDSource" anchor="true" person="false" />
		<!-- The @person attribute is by default false. -->
		<metadata value="CHANGE_ME" type="CatalogIDDigital" anchor="true" />
		
		<metadata value="CHANGE_ME" type="CurrentNoSorting" anchor="false" volume="true" />
		<!-- The @anchor attribute is by default false. -->
		<metadata value="zeitungen#livb" type="SubjectTopic" volume="true" />
		<metadata value="pt_zeitung" type="Publikationstyp" volume="true" />
		<metadata value="zeitungsherausgeber#livb" type="Classification" volume="true" />
		<metadata value="eli" type="ViewerInstance" volume="true" />
		
		<!-- Use @var to generate the final metadata values in the run. -->
		<metadata var="YEAR" value="Newspaper Volume _YEAR_" type="CatalogIDDigital" volume="true" />
		<metadata var="YEAR" value="Liechtensteiner Volksblatt (_YEAR_)" type="TitleDocMain" volume="true" />
		<metadata var="YEAR" value="_YEAR_" type="CurrentNo" volume="true" />
		<metadata var="YEAR" value="_YEAR_" type="PublicationYear" volume="true" />
	</set>
	
</config_plugin>
Parameter
Explanation

importFolder

This parameter is used to specify the directory from which the data is to be imported.

workflow

This parameter defines the name of the Goobi process template on the basis of which the processes are to be generated.

processtitle

Specify here what the title of the processes to be created should be. The year is added when the processes are created (e.g. New_York_Times_123456789).

pageNumberPrefix

If the pages in the Arabic pagination are to be preceded by a prefix, this can be defined here (e.g. "Page").

languageForDateFormat

Specify the language to be used for generating the output titles (e.g. en or de).

issueTitlePrefix

If a prefix is to be placed before the detailed date as the title of the newspaper issue, this can be entered here (e.g. "Issue from").

deleteFromSource

If the files to be imported are to be deleted from the import directory after the import, this can be specified here.

metadata

These elements can be used to specify which metadata should be used at newspaper and volume level for the processes to be created. An independent metadata is created from each element specified here. It accepts six attributes, whereby value and type are mandatory, while var, anchor, volume and person are optional. Further details can be found in the comments within the sample configuration.

https://github.com/intranda/goobi-plugin-workflow-newspaper-pages-importer
The plugin cannot be used without correct authorisation
Correctly assigned role for users
Open plugin for importing newspaper editions
User interface after performing the import
Generated newspaper editions with the associated metadata