Recreating Playground

From Digital Scholarship Group
Revision as of 15:13, 2 March 2021 by Sbauman (talk | contribs)
Jump to navigation Jump to search

Installing CentOS and WWO onto playground

This page is notes for re-creating the WWO environment from RH 6.10 (wwp.neu.edu) to CentOS 8.2 (playground). It will serve as a guide to start, and then hopefully a useful description for future similar upgrades.

CentOS

Installing CentOS was essentially trivial. We used the CentOS-8.2.2004-x86_64-boot.iso.

It appears that yum has been superceded by dnf.

Things we should or have installed

  • Emacs — installed 2020-11-30 with sudo yum install emacs-nox.x86_64
  • Emacs/nxml — was automagically installed by above (and has nice colors! :-)
  • make — installed 2020-11-30 with sudo yum install make.x86_64
  • Subversion — installed 2020-11-30 with sudo yum install subversion.x86_64
  • libxml2/xmllint — libxml2-2.9.7-7.el8.x86_64 is already installed
  • libxslt/xsltproc — installed 2020-11-30 with sudo yum install libxslt.x86_64
  • ant — intsalled 2020-11-30 with sudo yum install ant.noarch
  • screen — nope, Red Hat recommends tmux instead — installed 2020-11-30 with sudo yum install tmux.x86_64
  • Tomcat — no yum package found
  • git — installed 2020-11-30 with sudo yum install git.x86_64 (installed 46 dependencies, mostly Perl packages)
  • eXist — no yum package found, so see detailed description below
  • MariaDB (maybe? maybe not)
  • CouchDB (maybe? maybe not)
  • lessc (pkg = node-less) — no yum package found
    • Ash checked, they recommend installing via npm
    • npm — installed 2020-12-02 with sudo dnf install npm.x86_64
    • installed lessc with sudo npm install less -g
  • Perl — v 5.26.3 already installed
  • PHP — installed 2020-11-30 with sudo yum install php.x86_64
  • Python — I do not know if we want version 3.6 or 3.8 (or 2.7, which I doubt); see the RHEL8 doc
  • Awstats —not found
  • locate — already installed, but need to generate and cronify udpatedb; done 2020-11-30, I think. Ran sudo /usr/bin/updatedb and now locate seems to work. Symlinked /usr/bin/updatedb to /etc/cron.daily/updatedb. Hope that works. I think we check by loooking at timestamp of /var/lib/mlocate/mlocate.db.
  • gcc — installed 2020-12-02 with sudo dnf install gcc.x86_64
  • apxs — installed 2021-02-23 with sudo dnf install httpd-devel
  • libtool — installed 2021-02-23 with sudo dnf install libtool

Apache setup, modules, etc.

We (Syd & Ash) worked on configuring the web server for awhile, but were still having problems. Ernesto (user=nestochan) gave us a hand, performed some firewall magic, and now it is working. Need to find out what he did.

  • server side includes
  • whatever we need for security and textbaseIP stuff
    • mod_jk
    • ✓ /etc/httpd/passwd/
  • redirection stuff (mostly part of configuration)
  • probably want to check several files in wwp-test:/etc/httpd/ into version control; consider:
    • conf.d/webalizer.conf
    • conf.d/limitipconn.conf
    • conf.d/reqtimeout.conf
    • conf.d/wwp-blog.conf
    • conf.d/wwp.conf-ssl.conf
    • conf.d/mod_security.conf
    • conf.d/dsc-wiki.conf
    • conf.d/dsg-wiki.conf
    • modsecurity.d

files from /etc/httd/ whose differences between wwp and wwp-test need to be investigated

  • Files wwp-test/httpd/conf/httpd.conf and wwp/httpd/conf/httpd.conf differ
  • Files wwp-test/httpd/conf.d/awstats.conf and wwp/httpd/conf.d/awstats.conf differ
  • Only in wwp-test/httpd/conf.d: dsc-wiki.conf
  • Only in wwp-test/httpd/conf.d: dsg-wiki.conf
  • Files wwp-test/httpd/conf.d/limitipconn.conf and wwp/httpd/conf.d/limitipconn.conf differ
  • Files wwp-test/httpd/conf.d/mod_security.conf and wwp/httpd/conf.d/mod_security.conf differ
  • Files wwp-test/httpd/conf.d/redirects.conf and wwp/httpd/conf.d/redirects.conf differ
  • Files wwp-test/httpd/conf.d/ssl.conf and wwp/httpd/conf.d/ssl.conf differ
  • Only in wwp-test/httpd/conf.d: useless
  • Files wwp-test/httpd/conf.d/wwp-blog.conf and wwp/httpd/conf.d/wwp-blog.conf differ
  • Files wwp-test/httpd/conf.d/wwp.conf and wwp/httpd/conf.d/wwp.conf differ
  • Files wwp-test/httpd/conf.d/wwp.conf-ssl.conf and wwp/httpd/conf.d/wwp.conf-ssl.conf differ
  • Files wwp-test/httpd/conf.d/wwprd.conf and wwp/httpd/conf.d/wwprd.conf differ
  • Files wwp-test/httpd/conf.d/wwprewrite.conf and wwp/httpd/conf.d/wwprewrite.conf differ

other things to be moved over

  • WWO
  • WWP website
    • including configuration files
    • include git repos per /var/www/html/WWP/lab/README-WWP
  • eXist applications (see WWP_eXist-DB_applications)
  • either Awstats or something better
    • remember, /usr/local/bin/rgxg is used by Awstats
  • WordPress
  • /usr/local/bin/wwpwwwup.bash

things to be moved only to wwp-test

  • MediaWiki
  • CouchDB and the data produced for Cultures of Reception

things we need help with

  • MediaWiki (Karl)
  • WordPress (Karl)
  • https: access part of Apache config (Ernesto)
  • documenting what was done to firewall (Ernesto)
  • mod_jk?

Version control plan

Our plan is to keep all the Apache2 configuration stuff (/etc/httpd/) under Subversion on playground, wwp-test, local machines, etc.; BUT not on production (wwp) itself. To update that machine you will have to copy files over by hand. This is in part to help reduce chance of messing things up with an svn up and in part as an acknowledgement that a) karl & Ernesto work in this dir, too, and b) some of the files actually have to be different (due to name of host being different, i.e. wwp vs wwp-test vs 129.10.107.233).

eXist-DB

wget the JAR file from [1]. WWP is currently using eXist v4.7.1, but will soon be upgrading to 5.x. In v5.x, the configurations below are mostly the same, except most of the files are now stored in $EXIST_HOME/etc. 

useradd --home-dir /opt/local/eXistDB --create-home --system existdb
chmod 0775 /usr/local/eXistDB
sudo su - existdb

Add environmental variables to .bash_profile: 

export EXIST_HOME=/opt/local/eXistDB/eXist-current
export RUN_AS_USER=existdb

Then, in the terminal: 

source ~/.bash_profile
/etc/alternatives/jre_1.8.0/bin/java -jar /tmp/exist-4.5.0-setup.jar -console

As part of the install process, you'll be asked for a directory path for the newly-installed database. Use /opt/local/eXistDB/eXist-VERSION , where VERSION is the version number. Set the administrator password rather than leaving the field blank.

Create a symlink: ln -s eXist-VERSION eXist-current.

Navigate into eXist-current, aka $EXIST_HOME.

Jetty settings

Change the eXist port number, since 8080 is already in use on the WWP servers.

  • Create a backup of $EXIST_HOME/tools/jetty/etc/jetty-http.xml.
  • Change the HTTP port from 8080 to 8088.

Increase the maximum file size accepted for upload. (The default is 200,000.)

  • Create a backup of $EXIST_HOME/tools/jetty/webapps/exist-webapp-context.xml
  • Add a new property as a child of <Configure>:
<Set name="maxFormContentSize">400000</Set>

Webapp settings

Keep guest users from executing arbitrary XQueries. See Mathias Göbel's 2017-03-11 post on Exist-open, "get parameter _query = open proxy".

  • Create a backup of $EXIST_HOME/webapp/WEB-INF/web.xml.
  • Set //init-param[param-name = ('xquery-submission', 'xupdate-submission')] to "authenticated".

Database configuration

  • Create a backup of $EXIST_HOME/conf.xml
  • Set //indexer/@preserve-whitespace-mixed-content to "yes".
  • Set //serializer/@indent to "no".
  • Disable execution of arbitrary XQueries: set //builtin-modules/module[@uri eq 'http://exist-db.org/xquery/util']/parameter[@name eq 'evalDisabled']/@value to "true".
  • Add Saxon settings to //transformer[@class eq 'net.sf.saxon.TransformerFactoryImpl']:
<attribute name="http://saxon.sf.net/feature/recoveryPolicyName"
           value="recoverWithWarnings"
           type="string"/>
<attribute name="http://saxon.sf.net/feature/strip-whitespace"
           value="none"
           type="string"/>

At this point, if you can log into the eXist-DB Dashboard from the web, install the eXist-DB webapps. When the persistent scheduler has been installed, return to conf.xml and add this job to //scheduler: 

        <!-- Reschedule XQuery jobs after a restart. -->
        <job type="user" name="persistent-scheduler" 
             xquery="/db/apps/persistent-scheduler/rescheduler.xq"
             period="5000" repeat="0" />

The next time eXist starts up, the changes made in all these files will be applied.

Install eXist as a service

eXist comes with YAJSW, which acts as a wrapper daemon to run eXist automatically on server boot. YAJSW configuration files and scripts can be found at $EXIST_HOME/tools/yajsw.

Shut down eXist before installing the daemon.

  • Create a backup of $EXIST_HOME/tools/yajsw/conf/wrapper.conf.
  • Change "wrapper.app.account" to existdb\\existdb.

Previous installations have also changed "wrapper.java.command", "wrapper.console.title", "wrapper.ntservice.name", and "wrapper.daemon.pid.dir", but they aren't essential.

As root: 

EXIST_HOME=/opt/local/eXistDB/eXist-current RUN_AS_USER=existdb /opt/local/eXistDB/eXist-current/tools/yajsw/bin/installDaemon.sh

You may have the option to run the eXist service through "non-privileged systemd" or "privileged systemV-init". Playground currently (2020-12-04) uses systemd. Not sure what previous installations have used.

eXist v5.2.0 as a service

Newer versions of eXist don't require a wrapper at all. Use the instructions in the eXist documentation.


Progress Notes

November 2019

“playground” is the name Ashley & Syd give to the box they are using for experimentations. (Currently for web stat packages.) This page describes a procedure for installing a CentOS (i.e., RedHat clone) OS on it quickly.

  • Remember to back up iptables and restore it
  • We re-set the clock with instructions from a website we found.


2021-02-03

We have the apache2 web server up and running; loading http://129.10.107.233/WWP/ works in that data is loaded. However SSIs are not working. (Tomcat and mod_jk are not working yet either, but we did not expect them to.)

Next step: Figure out how to configure locally so that SSIs work. We tried moving in the old /etc/httpd/conf/httpd.conf file into the new environment and the entire web server failed to start up.


2021-02-08

We copied the Apache2 configuration files from the SVN trunk (read: following WWP Test configuration). Restarting the server failed due to missing modules in /etc/httpd/modules. That path is a symlink to ../../usr/lib64/httpd/modules, and the contents differ between Playground and Test.

We will need to figure out how to install some of the missing modules:

  • mod_limitipconn or its equivalent in Apache 2.4
  • mod_perl

We may need mod_fcgid (FastCGI). We don’t think we need mod_auth_mysql, mod_autoindex, or mod_wsgi (Python).


2021-02-09

We discovered that Playground had a date of 2018-07, and reset it to the correct date with sudo date --set. That also allowed us to install mod_limitipconn and mod_perl, since those failed due to a problem.

Continuing with our Apache configuration test, we found that we need an SSL certificate in order to fully test the server setup (for this wiki, for example).

Ernesto installed the SSL certificate from WWPDev, so we can continue working on Apache configuration with apachectl configtest.

We also created a symlink from /var/lib/tomcat9 (correct?) to the /usr/local/share/apache-tomcat.../webapps directory. Tomcat still needs to be set up as a service to run on server restart.


2021-02-23

We’ve installed the Apache module mod_jk from source (mod_jk connects the server to Tomcat). We found and followed instructions here, though we couldn’t start httpd in step one. The JkMount directives now seem to work with the new mod_jk.so file installed at /etc/httpd/local-modules.

We fixed the paths to the SSL certificate provided by Ernesto.

There are some warnings remaining, and expected paths and/or permissions that are missing in our setup.


2021-02-26

We continued debugging our Apache2 configuration. We discovered that our newly-compiled mod_jk.so module had to have the same SELinux policy as httpd. We used the chcon command, with the "reference" argument to copy the policy from httpd to the module.

We also created the /var/log/apache2 directory, and added READMEs in that directory and /var/log/httpd, explaining what should be stored in each.

After these changes, the Apache configuration syntax came back OK.

The httpd service still fails to start, however, and there don’t seem to be clues in the command line output of [?].

We checked the /var/log/apache2/error_log, discovered debugging and info messages about the security certificate. We updated the VirtualHost names to match playground’s, which resulted in a warning in the error log: the server name does not match the names in the security certificate. We expected this, and it really shouldn’t affect the server starting up.

Next step is to discover why Apache still won’t run.

2021-03-02

On the thought that the reason the webserver will not start might be because of the attempt to load modules multiple times, we removed all duplicate LoadModule calls. To see what it was before we removed these, check out 42719 or later, but before 42805.

The good news is we managed to remove all the duplicate module warning messages. The bad news is that (kinda as expected) it did not change the fact that the webserver will not start, nor the (kinda useless) error messages in the log file.

Retrieved from "https://dsg.northeastern.edu/wiki/index.php?title=Recreating_Playground&oldid=3774"