February

Goobi viewer Digest for February 2022

Coming soon 🚀

Display of a disclaimer for records
Comment views
Upload of records

Announcements 🇺🇦

This Digest comes at a time when war has broken out in Europe. We are receiving horrific news from Ukraine.

We stand in solidarity with all the people affected by this terrible war.

Developments

The Goobi viewer can now display a cookie banner. This is configured in the backend. The functionality can be switched on or off there and the text for the banner can be managed. In addition, a list of CMS pages is available on which the banner should not be displayed. This may be necessary, for example, for an imprint, a privacy policy or a cookie policy page. Furthermore, it is possible to reset the consent of users, for example if there has been a change in the text and renewed consent is necessary.

The cookie banner is linked to the Matomo tracking functionality. If both are active, tracking only takes place when the text displayed in the banner has been accepted. If the text is not accepted, tracking is prevented.

The information about acceptance or rejection of the text is stored in the browser's local storage. This means that it is valid beyond the session and the text only has to be accepted once.

Database

In the Goobi viewer we have had a very simple but effective principle in development for many years:

Bug fix before feature

This means that when we see or are notified of a bug in the software, it is fixed before we continue with the development of new features. The bug fixes are then incorporated into the current code base and - if possible and necessary - we also backport the specific fix into earlier versions.

In practice, these are often only issues of half an hour or a few lines of source code. That's all it often takes, and it's easy to schedule the work at the beginning or end of the day.

This month we had a different kind of error. The Goobi viewer lost its database connection out of the blue. From one moment to the next, when a page was opened, the message "Connection has already been closed" was displayed. The only thing that helped was to restart the Tomcat process completely.

This error puzzled us all, because the Goobi viewer does not manage the database connection itself, but the Tomcat does and only passes it on. The Tomcat, in turn, always had enough open and unused connections, even when the Goobi viewer gave the error message, the connections were still active. To make matters worse, we had absolutely no way to reproduce or provoke the error ourselves. We could only see the symptom on installed live systems. It was not possible to recreate the symptom in the development environment. The symptom sometimes appeared five minutes after starting the Tomcat and then again only after two days. Without any discernible differences in use...

Various colleagues then started searching. First in the source code to see if any connections were accidentally closed. During this search, bugs were found and fixed, but none that were specifically related to this. The analysis was then extended to other parts in the chain: Does the error only occur with MariaDB or also with MySQL? Does the error no longer occur with older or newer Tomcat versions? Does the error occur when the connector that Tomcat uses to connect to the database is changed? What happens if we change the parameters (unchanged for years) that Tomcat uses to connect to the database? All approaches were tested and with each change we had to check on the installed systems whether it would lead to success. Unfortunately, the result was always sobering: all measures did not lead to any change. At a certain point, we could say with a high degree of certainty that no error in the code was closing the connection and that the setup and the components in between were not the cause. But the error still existed.

In the meantime, we had activated logging, which resulted in several GB of log files per hour. This meant that everything the Goobi viewer did with the database was logged. Here is an example of logging a simple SQL query:

[EL Finer]: connection: 2022-02-24 05:01:56.878--ServerSession(1137386215)--Thread(Thread[ajp-nio-0:0:0:0:0:0:0:0-8009-exec-1,5,main])--client acquired: 33039885
[EL Finer]: transaction: 2022-02-24 05:01:56.878--ClientSession(33039885)--Thread(Thread[ajp-nio-0:0:0:0:0:0:0:0-8009-exec-1,5,main])--acquire unit of work: 964015310
[EL Finest]: query: 2022-02-24 05:01:56.879--UnitOfWork(964015310)--Thread(Thread[ajp-nio-0:0:0:0:0:0:0:0-8009-exec-1,5,main])--Execute query ReadAllQuery(referenceClass=CMSStaticPage sql="SELECT static_page_id, cms_page_Id, static_page_name FROM cms_static_pages WHERE (cms_page_Id = ?)")
[EL Finest]: connection: 2022-02-24 05:01:56.879--ServerSession(1137386215)--Connection(1350487645)--Thread(Thread[ajp-nio-0:0:0:0:0:0:0:0-8009-exec-1,5,main])--Connection acquired from connection pool [read].
[EL Finest]: connection: 2022-02-24 05:01:56.879--ServerSession(1137386215)--Connection(1350487645)--Thread(Thread[ajp-nio-0:0:0:0:0:0:0:0-8009-exec-1,5,main])--reconnecting to external connection pool
[EL Fine]: sql: 2022-02-24 05:01:56.879--ServerSession(1137386215)--Connection(2077776754)--Thread(Thread[ajp-nio-0:0:0:0:0:0:0:0-8009-exec-1,5,main])--SELECT static_page_id, cms_page_Id, static_page_name FROM cms_static_pages WHERE (cms_page_Id = ?)
        bind => [60]
[EL Finest]: connection: 2022-02-24 05:01:56.879--ServerSession(1137386215)--Connection(1350487645)--Thread(Thread[ajp-nio-0:0:0:0:0:0:0:0-8009-exec-1,5,main])--Connection released to connection pool [read].
[EL Finer]: transaction: 2022-02-24 05:01:56.879--UnitOfWork(964015310)--Thread(Thread[ajp-nio-0:0:0:0:0:0:0:0-8009-exec-1,5,main])--release unit of work
[EL Finer]: connection: 2022-02-24 05:01:56.879--ClientSession(33039885)--Thread(Thread[ajp-nio-0:0:0:0:0:0:0:0-8009-exec-1,5,main])--client released

After several colleagues had been staring rhythmically at these growing log files for several days, new questions were formulated and researched. On Tuesday morning, there was a new approach: Does the way of implementation have to be different for each individual database connection? This has always existed in exactly this way, but perhaps it does not conform to best practice? At least there were new ideas after reading documentation and tutorials on this.

After a short discussion, we decided to go this way. In the meantime, several colleagues were convinced that such a change would definitely be an improvement of the code, even if it is not possible to say 100% if the previous implementation was "wrong".

Many, many hours of work and several thousand lines of modified code later, the time had come and we were able to test a -SNAPSHOT on the systems that reliably continued to lose the connection randomly at undefinable intervals. From then on we had to wait and hope...

In the end, exactly this change turned out to be the solution. All systems on which we have installed the -SNAPSHOT version have not lost the connection since then. The graphs in the monitoring have remained completely unchanged.

It was not always easy to stay true to the principle of "Bug fix before feature" last month. Our release planning and roadmap has been significantly postponed as a result, because if we now add up the times of all the colleagues involved who have worked on the problem, it is almost a complete month of development time.

CMS

CMS pages can now be set as the start page for a work. This makes it possible when opening works via the RSS feed, the search hit list or various resolvers to be directed directly to a CMS page that contains further information on the work.

This functionality is available if a CMS page has the option "Linked work" activated together with the option "Use as default view for linked work".

Search

Up to now, the values in the facets were always translated. This is necessary, for example, to display not the internal value "lat" but its translation "Latin".

However, there are cases where this behaviour is not desired. For example, if the keywords used are in English and should also be visible as such. Then the value "work" would be translated as "work", although "work" is meant in terms of content.

For this reason, there is now a new switch in the configuration file where the translation of values in selected facets can be optionally deactivated.

Backend

Some of you may have already noticed the small indicators in the sidebar on the screenshots above. These are now available for the "Access Licences" and "Translations" page and indicate potential tasks in the areas.

Version numbers

The versions that must be entered in the pom.xml of the theme in order to get the functions described in this digest are:

<dependency>
    <groupId>io.goobi.viewer</groupId>
    <artifactId>viewer-core</artifactId>
    <version>22.02</version>
</dependency>
<dependency>
    <groupId>io.goobi.viewer</groupId>
    <artifactId>viewer-core-config</artifactId>
    <version>22.02</version>
</dependency>
<dependency>
    <groupId>io.goobi.viewer</groupId>
    <artifactId>viewer-connector</artifactId>
    <version>22.02</version>
</dependency>