WWP eXist-DB applications

The Women Writers Project uses the eXist-DB database to query, transform, and publish XML files. Most of these “applications” are actually divided between eXist and the WWP server file system. In general, eXist is used to store and index XML files. The database also holds XQuery scripts which, on request, parse and return data in a web-friendly format (generally JSON or HTML).

On the server file system, an application mainly consists of an index HTML file and a Javascript file, as well as other web assets (CSS, images, etc.). The Javascript file is a Backbone.js application that makes calls out to the eXist application, using the responses to populate most pages in the browser. In fact, though the URL changes, a user navigating the WWP subproject site never actually leaves index.html. Instead, the Backbone app modifies the webpage and browser URL to match the conceptual “page” chosen by the user.

Prerequisite libraries and applications

Many of the WWP “endeavor”-specific applications are built upon a common codebase, in the form of EXPath packages. These should be installed before the other applications.

Name	Description	EXPath app?	Filepath	Database path
WWP Function Library	The most common set of functions. The package includes a shim for JSON-related functions.	Yes	WEB/utils/lib	N/A (libraries are imported by URI)
API Library	Functions to simplify the creation of APIs.	Yes	WWP Public Code Share	N/A (libraries are imported by URI)
eXist triggers	Functions that can help specific applications do things in response to a database action (such as, change the permissions of a file uploaded to this folder).	No	WEB/utils/stylesheets/exist-triggers	/db/exist-triggers.xql
Persistent Scheduler	A library and small application that reschedules XQuery cron jobs on request. eXist should be configured to run the rescheduling script after startup.	Yes	NEU-DSG GitHub repository	/db/apps/persistent-scheduler

eXist-supported endeavors

WWP endeavors which have an eXist-DB component are:

Women Writers in Context

The code for the WWiC website can be found in Subversion at DEV/exhibits. (Unlike the other endeavors, WWiC has no separation of code from data.) On the server, the index page and assets are symlinked from the “exhibits” directory to WEB/context. As a result, WWiC is visible from both https://wwp.northeastern.edu/context and https://wwp.northeastern.edu/research/projects/exhibits.

WWiC is not an EXPath application, but it does use XQueries as REST endpoints. In eXist, the XML, XSLT, and XQueries can be found at /db/exhibits.

eXist users in the “exhibits” group should have write access to the collection on the WWP-Test server. This enables easy pre-publication proofing. Publication on the production server requires a database admin.

Women Writers in Review

The code for the WWiR website can be found in Subversion at DEV/reception/utils. On the server, the index page and assets are symlinked from the “reception” directory to WEB/review. Unlike WWiC, Review is available only at https://wwp.northeastern.edu/review/index.html. The files are stored at https://wwp.northeastern.edu/research/publications/reception/utils/index.html, but the Backbone.js script will not work if accessed there.

WWiR is also an EXPath package, and uses RESTXQ to serve out data via an API. The XML files are stored in eXist at /db/reception-data, and the EXPath application will be installed at /db/apps/review. The base URI for the API is “http://www.wwp.northeastern.edu/exist/restxq/reception .” For more information on the API endpoints available, see https://www.wwp.northeastern.edu/exist/restxq/reception/api-docs.

Featured Quotation

The Featured Quotation on the WWP home page is generated randomly, with a request to eXist from the Javascript file at WEB/utils/bin/javascript/wwp.js. The HTML response then populates the quote widget.

The code in eXist is not an EXPath application. The XQuery qotr.xq and XSLT qotd.xslt are maintained in WEB/utils/stylesheets, and are saved in eXist at /db/quotes. Unlike most other eXist endeavors, the XML source is not stored in eXist. Instead, the XQuery uses the quotography on the appropriate server (production by default) at WEB/utils/xml/quotography.xml.

Text Tracker

The Text Tracker is a small EXPath application that uses data from the Text Tracking Trello board to create a table of forthcoming WWO texts. The code is maintained at DEV/text-tracker. When installed into eXist, the XQuery and XSLT will be stored at /db/apps/text-tracker, and the collection /db/trello-data will be created, if it doesn't already exist.

Important: Because the Text Tracker requires a Trello user's credentials, the file at /db/trello-data/trello.xml must be readable by only database administrators. It is not checked into Subversion. The XML structure of trello.xml looks like this:

<credentials>
  <key></key>
  <oauthSecret></oauthSecret>
  <token as="USERNAME"></token>
</credentials>

When the “forthcoming texts” web page loads, Javascript sends an AJAX request to get the table from the EXPath application. The Javascript file can be found at WEB/utils/bin/javascript/forthcoming.js.

See Trello's documentation for more on its API.

Internal Documentation

The code for Internal Documentation is located in Subversion at WEB/research/publications/documentation/internal.

WWiC is not an EXPath application, but it does use XQueries as REST endpoints. In eXist, the XML, XSLT, and XQueries can be found at /db/wwpdoc.

eXist users in the group “wwpdoc” should have write access to this collection on the WWP-Test server. This enables easy pre-publication proofing. Publication on the production server requires a database admin.

Inspectre

Unlike the other eXist-endeavors, the Inspectre lives only in eXist. It is an EXPath application maintained at DEV/inspectre, and installed into eXist at /db/apps/inspectre. As part of the installation process, missing XML reports in /db/inspectre-data/current are added from DEV/inspectre/reports. However, once a report is present in the Inspectre data collection, it will not be replaced during installation. This keeps encoders’ edits from being overwritten.

The Inspectre should be installed only on WWP-Test. eXist users within the “inspectre” group will be able to act on reports.

Maintaining eXist-supported endeavors

Updating the HTML content of a web page

To update the content of any given page, you will need to determine if that content is generated, and, if it is generated, where it is generated. This varies from project to project.

In general: representations of XML documents are generated with XSLT stored in eXist. You may need to update the XML encoding, the XSL transformation, or both. The HTML output is usually passed on to XQuery and Javascript without changes.

The outer structure of a web page, such as the header or footer, is likely generated by Backbone.js. Whole pages of static content, such as “About” pages, can be maintained in HTML files, Backbone.js templates, or XQuery functions.

Updating images, CSS, Javascript

In general, web assets—including images, CSS stylesheets, and Javascript files—are maintained on the server file system, not in eXist. To add or update these files, commit your changes to Subversion. Then log into the WWP server using your Centrify credentials, and cd to the directory where those assets are housed. Issue an svn up, then check the website to make sure the changes went through. (Ideally you’ll have updated Test first, so that any bugs can be resolved without downtime on production.)

Updating XML data

If XML data is stored in eXist for transformation and publication, it can be updated by navigating to the eXide web editor, logging in, and uploading the newest files to the correct data folder.

Besides using a direct link, you can also navigate to eXide through the eXist Dashboard “Launcher”:

The eXide editor looks like this. You’ll need to log in before adding or updating any files.

eXide has a feature called “DB Manager”, which you can find under the “File” menu. Once you’ve logged in, you can use the DB Manager to upload and modify files within the eXist database. Most directories can only be edited by users with certain permissions. If you find you don’t have the ability to work in a folder you think you should have access to, contact the person in charge of WWP’s eXist instances (currently Ash Clark).

Adding-files-to-eXist 2021-01-19 03-manage-database.png

As an example, say you want to add a new Exhibit to Women Writers in Context. You would commit your changes as usual, then log into eXide. Once in the DB Manager, you would navigate to the directory you want. Fair warning: the DB Manager is set up so that clicking the contents of the “Name” field will start the process of changing the name of the file or directory. To navigate the database, double-click on one of the other table cells in the row you want.

The Exhibits XML files are stored at /db/exhibits/xml. Once you’d navigated to the “exhibits” and then “xml” directories, you would start the upload. The button to do this is the one shaped like a cloud with an arrow, found at the top of the DB Manager widget.

Adding-files-to-eXist 2021-01-19 05-upload-file.png

After you’ve changed something in eXist, you can visit the relevant WWP Test application in order to proof the file in its HTML form. You can continue to edit the file in Subversion and re-upload it to eXist as often as you like.

Updating XQuery or XSLT

If the eXist part of an endeavor is not an EXPath package, updating its XQuery or XSLT is as easy as committing your changes, then uploading the changed file to eXist using the steps above.

If the project does include an EXPath package, you will first need to rebuild the package. On the command line, navigate to the project-specific folder where the code is kept. In the directory containing a file called “build.xml”, use the command ant. Apache Ant should then create a XAR file (really a ZIP file).

To install an EXPath package into eXist, log into the Dashboard. (You must have database administrator privileges!) Click on the icon for the “Package Manager” app.

Error creating thumbnail: File missing

When opened, the Package Manager shows a list of the packages that are currently installed, with those able to be upgraded at the top. If you click the upload icon, a pop-up menu will appear, allowing you to install the XAR file you just created.